Smith was born in Utica, Oneida County, New York, to Peter Gerrit Smith (1768-1837) and Elizabeth (Livingston) Smith (1773-1818), daughter of Col. James Livingston (1747-1832) and Elizabeth (Simpson) Livingston (1750-1800).
About 1828 Smith became an active temperance campaigner, and in his hometown of Peterboro, he built one of the first temperance hotels in the country.