 Lady Elizabeth was born in May 1663 at Bretby, Derbyshire, the daughter of Philip Stanhope, 2nd Earl of Chesterfield, and Lady Elizabeth Butler, eldest daughter of James Butler, 1st Duke of Ormonde.Her paternity was in doubt as despite her putative father's own licentious conduct; he had been the lover of the notorious Barbara Villiers, mistress of King Charles II of England.The earl believed that Elizabeth was not his daughter.