William was born in London in 1924 to Charles Harvey-Kelly (Lieutenant Colonel in the Indian Army, Baluch Regiment) and Sybil (medical doctor and daughter of a Liberal MP).
He was originally educated at Castle Park School in Dublin, the Wellington College, Berkshire eventually studying law at University College Oxford.