In the early 1850s, there was considerable hostility toward immigrants, especially Catholics, who had recently arrived in the United States in large numbers, and several nativist organizations, including the Order of the Star Spangled Banner, sprang up in reaction. By 1854 the order had morphed into the American Party, which became known as the Know Nothings. (In its early days, members were sworn to keep its internal deliberations private and, if asked, were to say they knew nothing about them.)[115]
Many from Fillmore's "National Whig" faction had joined the Know Nothings by 1854 and influenced the organization to take up causes besides nativism.[116] Fillmore was encouraged by the success of the Know Nothings in the 1854 midterm elections in which they won in several states of the Northeast and showed strength in the South. On January 1, 1855, he sent a letter for publication that warned against immigrant influence in American elections, and he soon joined the order.[117]
Later that year Fillmore went abroad, and stated publicly that as he lacked office he might as well travel. 