Fergusson was born on 6 May 1911, the third son and fourth child of Sir Charles Fergusson, 7th Baronet and his wife Lady Alice Mary Boyle, a daughter of David Boyle, 7th Earl of Glasgow.
Fergusson married Laura Margaret Grenfell (born 14 April 1920, died 1979) on 22 November 1950 and they had one child, Hon George Duncan Raukawa Fergusson (born 30 September 1955)[REF] who served as British High Commissioner to New Zealand from 2006 to 2010 and Governor of Bermuda from 2012.