[165] Tyler retired to a Virginia plantation, originally named Walnut Grove (or "the Grove"), located on the James River in Charles City County. He renamed it Sherwood Forest, in a reference to the folk legend Robin Hood, to signify that he had been "outlawed" by the Whig Party.[166] He did not take farming lightly and worked hard to maintain large yields.[167] His neighbors, largely Whigs, appointed him to the minor office of overseer of roads in 1847 in an effort to mock him. To their displeasure, he treated the job seriously, frequently summoning his neighbors to provide their slaves for road work, and continuing to insist on carrying out his duties even after his neighbors asked him to stop.[168]
The former president spent his time in a manner common to Virginia's First Families, with parties, visiting or being visited by other aristocrats, and spending summers at the family's seaside home, "Villa Margaret".[169] In 1852, Tyler happily rejoined the ranks of the Virginia Democratic Party and thereafter kept interested in political affairs.[165] However, Tyler rarely received visits from his former allies and was not sought out as an adviser. 