Fillmore rarely spoke about the immigration question, focused on the sectional divide, and urged the preservation of the Union.[122][123]

Fillmore won only Maryland (pink).
Once Fillmore was back home in Buffalo, he had no excuse to make speeches, and his campaign stagnated through the summer and the fall of 1856. Political fixers who had been Whigs, such as Weed, tended to join the Republican Party, and the Know Nothings lacked experience at selling anything but nativism. Accordingly, Fillmore's pro-Union stance mostly went unheard. Although the South was friendly towards Fillmore, many people feared that a Frémont victory would lead to secession, and some of those who were sympathetic to Fillmore moved into the Buchanan camp for fear of splitting the anti-Frémont vote, which might elect the Republican.[124] Scarry suggested that the events of 1856, including the conflict in Kansas Territory and the caning of Charles Sumner on the floor of the Senate, polarized the nation and made Fillmore's moderate stance obsolete.[125]
Buchanan won with 1,836,072 votes (45.3%) and 174 electoral votes to Frémont's 1,342,345 votes (33.1%) and 114 electoral votes. 