Robert Wilmot was born the natural son of Sir Robert Wilmot the first baronet of Osmaston Hall, the Secretary to the Lord Lieutenant of Ireland.
Wilmot remarried Mariana Howard in 1795, Mariana was the heiress of Charles Howard of Stafford.