Fraser was educated at Ampleforth College, Balliol College, Oxford, where he was President of the Oxford Union, and at the Sorbonne.
Fraser was elected Member of Parliament for Stone in 1945, later Stafford and Stone following constituency boundary changes, from 1950 until 1983, and then Stafford until his death.