[269] During his presidency, many abolitionists harshly criticized him as an instrument of the "Slave Power", and claimed that spreading slavery was the reason he supported Texas Annexation and later war with Mexico.[270] Polk did support the expansion of slavery's realm, with his views informed by his own family's experience of settling Tennessee, bringing slaves with them.[271] He believed in Southern rights, meaning both the right of slave states not to have that institution interfered with by the Federal government and the right of individual Southerners to bring their slaves with them into the new territory.[272] Though Polk opposed the Wilmot Proviso, he also condemned southern agitation on the issue, and he accused both northern and southern leaders of attempting to use the slavery issue for political gain.[273]
On March 4, 2017, new tombstones for three of his slaves, Elias Polk, Mary Polk and Matilda Polk, were placed in the Nashville City Cemetery. Elias and Mary Polk both survived slavery, dying in the 1880s; Matilda Polk died still in slavery in 1849, at the age of about 110.