 Born in Lochvale near Langholm, Smith was the son of Eaglesfield Smith and his wife Judith Ann Irving, daughter of Sir Paulus Aemilius Irving.He was educated at University College, Durham, receiving an M.A.in 1852.