 Baker was born in Petersham, Massachusetts, the son of Joseph Baker and Molly Stevens, and came to Dunham, Quebec with his parents in 1799.He was involved in raising and importing livestock.In 1818, he married Lavina Barnes, his cousin.