Born on 11 January 1930, in the Manor House, Bletchingley, Surrey, he is the younger brother of Nicolas Stacey.
At Eton College (1943-48) Stacey became a fourth-generation successive Stacey pupil at Eton, where he was a solo treble, the founder of Wotton's Society in the field of philosophy, editor (with Douglas Hurd) of the weekly Eton College Chronicle, winner of the Essay Prize, and House Captain.