Born at Moulsham, Essex, Mildmay was the fourth and the youngest son of Thomas Mildmay, Auditor of the Court of Augmentations for Henry VIII, by his wife, Agnes Read.
Mildmay was educated at Christ's College, Cambridge, but apparently failed to take his degree [REF].