William David Ormsby-Gore was born on 20 May 1918 in London, the second son of William Ormsby-Gore, 4th Baron Harlech, a Conservative politician, and Lady Beatrice Edith Mildred Gascoyne-Cecil.
After the war his father handed over to him all his land, and Ormsby-Gore farmed the 400 acres (1.6 km²) of the Woodhill Estate, Oswestry, Shropshire.