 Smith was born in Barre, Massachusetts to Deacon Samuel Smith and Patience Gregory Smith.His family moved to St. Albans in 1800, and he attended the common schools.Smith later studied law, first with his brother in law Roswell Hutchins, and later with Benjamin Swift.