 Prior to her birth, Samuel Pepys alleged that Elizabeth's mother, tired of her husband's neglect of her in his pursuit of Barbara Villiers, had had affairs with both James Hamilton and James, Duke of York, who would later rule as King James II of England.In fact, she was caught "in flagrante" with the latter.Historian Antonia Fraser describes Elizabeth's mother as having been one of the mistresses of the Duke of York.