 The property, at the base of Tobacco Row Mountain, was first owned by Duncan Graham of Caroline County, who sold to George McDaniel in 1788, who in turn sold to Philip Burton in 1799.After an inheritance dispute, the property, now , passed to John R. Irvine of Bedford, Virginia, who sold it in 1835 to Charles Ellis of Richmond.The land is believed to have been used to cultivate tobacco for Ellis' company, Ellis and Allan.