nearby in Albemarle County. Furthermore, the wife of James Madison, Dolley Payne Todd Madison, was Coles's first cousin, and Coles became a frequent guest at their Montpelier plantation also nearby. James Monroe owned Ash Lawn-Highland plantation on the other side of Green Mountain for 24 years (until forced to sell it in 1825 due to financial problems), and offered young Coles use of his library, although the relationship with this family was more distant since Monroe split his time at Oak Hill plantation in Loudoun County. Isaac managed and ultimately inherited Enniscorthy, subject to a life estate held by his mother, who died in 1826. His brother Walter had drawn his share of the inheritance early and managed Woodville plantation for many years before his father's death. John Coles III built a mansion called Estouteville on his inherited portion, and Tucker Coles built Tallwood plantation on the upper acres that he inherited—both married daughters of Sir Peyton Skipwith, the only baronet in Virginia. Their sister Rebecca became the second wife of South Carolina planter Richard Singleton, with whom she had five children. Her sister Elizabeth (Betsy) never married. Mary Coles married Robert Carter and moved to his nearby Redlands plantation. Their sister Emily married Richmond lawyer John Rutherfoord, who owned Tuckahoe plantation in Goochland County and later served as acting governor of Virginia (1841–42). Their son John Coles Rutherfoord witnessed the Civil War's devastation, and his grandson W.A.R. Goodwin became an Episcopal minister and helped found Colonial Williamsburg. Their sister Callie (Sarah) married Andrew Stevenson, who served as Speaker of the United States House of Representatives as well as American minister to the United Kingdom. Career Service in the White House Some months after taking office as President, James Madison invited Coles to become his private secretary. His brother Isaac had been performing those duties, as well as delicate courier tasks (particularly with France in the days leading up to the War of 1812). However, Isaac Coles, perhaps inappropriately acting out his earlier military career dreams, beat up Maryland representative Roger Nelson and, after a critical Congressional report concerning the incident, was forced to submit his resignation on December 29, 1809. Neighbor James Monroe (soon to become Secretary of State) convinced Edward Coles to accept that secretarial position, and Coles served from January 1810 to March 1815, despite intervals of ill-health. Coles' term as presidential secretary delayed his plans to