 Barrie was born at St. Augustine, Florida on 5 May 1774, the son of Scottish surgeon Dr Robert Barrie of Sanquhar and his wife, Dorothea (Dolly) Gardner, the sister of Sir Alan Gardner.His mother returned to England on the death of her husband in 1775, and settled in Preston, Lancashire.In 1784 she remarried George Clayton, a textile manufacturer, while her son was schooled at Neston, Cheshire, and later at Dedham.