[5]
Nathaniel Fillmore was the son of Nathaniel Fillmore Sr. (1739–1814), a native of Franklin, Connecticut, who became one of the earliest settlers of Bennington, Vermont, when it was founded in the territory that was then called the New Hampshire Grants.[6]
Nathaniel Fillmore and Phoebe Millard moved from Vermont in 1799 and sought better opportunities than were available on Nathaniel's stony farm, but the title to their Cayuga County land proved defective, and the Fillmore family moved to nearby Sempronius, where they leased land as tenant farmers, and Nathaniel occasionally taught school.[7][8] The historian Tyler Anbinder described Fillmore's childhood as "one of hard work, frequent privation, and virtually no formal schooling.”[4]

Historical marker at the site of Fillmore's birth in Cayuga County, New York
Over time Nathaniel became more successful in Sempronius, but during Millard's formative years, the family endured severe poverty.[b] Nathaniel became sufficiently regarded that he was chosen to serve in local offices, including justice of the peace.[11] Hoping that his oldest son would learn a trade, he convinced Millard, who was 14, not to enlist for the War of 1812[12] and apprenticed him to clothmaker Benjamin Hungerford in Sparta.