Fraser was the son of the 14th Lord Lovat (commonly known as the 16th Lord), and Laura, daughter of Thomas Lister, 4th Baron Ribblesdale.
He married Rosamond Broughton (1917-2012), the daughter of Jock Delves Broughton, on 10 October 1938, with whom he had six children [REF].