 The house was originally built as a four-room house about 1803 and was called Clergy Hall by Dr. James McElhenny, who was the pastor of Hopewell Presbyterian Church.The home later became the home of John C. Calhoun and his wife Floride Calhoun in 1825.Calhoun enlarged it to fourteen rooms and renamed it Fort Hill for nearby Fort Rutledge, which was built around 1776.