 Anne was the posthumous younger daughter of Sir Henry Lee, and a member of a wealthy family.Her mother died not long after her birth.She and her sister Eleanor were brought up at Adderbury House, where they lived with the mistress, mother and grandmother of its owner, the poet and libertine John Wilmot, 2nd Earl of Rochester, who was Anne Wharton's uncle.