[38] Despite Fillmore's support of the Second Bank as a means for national development, he did not speak in the congressional debates in which some advocated renewing its charter although Jackson had vetoed legislation for a charter renewal.[39] Fillmore supported building infrastructure by voting in favor of navigation improvements on the Hudson River and constructing a bridge across the Potomac River.[40]
Anti-Masonry was still strong in Western New York though it was petering out nationally. When the Anti-Masons did not nominate him for a second term in 1834, Fillmore declined the Whig nomination, seeing that the two parties would split the anti-Jackson vote and elect the Democrat. Despite Fillmore's departure from office, he was a rival for the state party leadership with Seward, the unsuccessful 1834 Whig gubernatorial candidate.[41] Fillmore spent his time out of office building his law practice and boosting the Whig Party, which gradually absorbed most of the Anti-Masons.[42] By 1836 Fillmore was confident enough of anti-Jackson unity that he accepted the Whig nomination for Congress. 