 Through her influence Buxton became associated with the Gurney family of Earlham Hall, Norwich.He was especially close to Joseph John Gurney, his sister the prison reformer Elizabeth Fry, and their sister Hannah, whom he married in May 1807.He lived at Northrepps Hall in Norfolk.