 He was born in Park Street, near Grosvenor Square, London, the fourth child of Robert Ferguson (1799–1865) and Mary Mcleod of Skye.His father was born in India, a close friend of Sir John Macpherson, Governor-General of India, and Sir Walter Scott.Robert was an eminent physician who also took an interest in insects, literature and other matters becoming Physician Extraordinary to Queen Victoria.