With an estimated 100 slaves[6] and a 5,000-acre (2,000 ha) plantation, Madison's father was among the largest landowners in the Virginia Piedmont.[8] In the early 1760s, the Madison family moved into a newly built house that they named Montpelier.[9] Madison grew up as the oldest of twelve children,[10] with seven brothers and four sisters, though only six lived to adulthood.[9] Of the surviving three brothers (Francis, Ambrose, and William) and three sisters (Nelly, Sarah, and Frances), it was Ambrose who would eventually help to manage Montpelier for both his father and older brother until his own death in 1793.[11] President Zachary Taylor was a descendant of Elder William Brewster, a Pilgrim leader of the Plymouth Colony, a Mayflower immigrant, and a signer of the Mayflower Compact; and Isaac Allerton Jr., a colonial merchant, colonel, and son of Mayflower Pilgrim Isaac Allerton and Fear Brewster. Taylor's second cousin through that line was Madison.[12]

Virginia historic marker for Birthplace of President James Madison in Port Conway, Virginia
From age 11 to 16, Madison studied under Donald Robertson, a Scottish instructor who served as a tutor for several prominent planter families in the South. 