Ben Smith's home was in Marylebone, London, but from 1816 he inherited and purchased property near Hastings: Brown's Farm near Robertsbridge, with an extant house built about 1700, and Crowham Manor, Westfield, which included 200acre.
On their return to Sussex they lived openly together at Brown's and had two more children.