Graham returned to Cumberland and began a programme of improving the family estate at Netherby, with the intention that such improvements would benefit both the landowner and the tenant.
On 8 July 1819, Graham married Frances ("Fanny") Callander, of Craigforth and Ardkinglas, a famous society beauty.