Rougier was the only child of novelist Georgette Heyer and her husband, George Ronald Rougier CBE QC, a mining engineer who later became a barrister.
He was educated at Marlborough College, and read classics and law at Pembroke College, Cambridge from 1952 to 1955.