The site was previously occupied by the fortress of Iestyn ap Gwrgant, lord of Glamorgan and the last ruler of the Welsh kingdom of Morgannwg; its fields are still known as Caer Wrgan ("Wrganstown").
Dimlands was leased to the architect Sir Matthew Digby Wyatt (1820-1877) in 1875, he being the husband of Mary Nicholl of The Ham, a great niece of Robert.