He married Ann Winston Carr, first cousin to Patrick Henry, in Spotsylvania County, Virginia, in December 1760, and their children were: Elizabeth Waller, b. 1762; Mary Winston Waller, b.
The plantation he owned later became known locally as both "Waller's Ford" and the "Village of Waller".