She raised her three children with Birley at Pelham cottage.
In 1964, she embarked on a decade-long extramarital affair with the entrepreneur Sir James Goldsmith, a member of the prominent Goldsmith family.