[5][6]
Taylor was a descendant of Elder William Brewster, a Pilgrim leader of the Plymouth Colony, a Mayflower immigrant, and a signer of the Mayflower Compact; and Isaac Allerton Jr., a colonial merchant, colonel, and son of Mayflower Pilgrim Isaac Allerton and Fear Brewster. Taylor's second cousin through that line was James Madison, the fourth president.[7] He was also a member of the famous Lee family of Virginia, and a third cousin once removed of Confederate General Robert E. Lee.[8]
His family forsook its exhausted Virginia land, joined the westward migration and settled near future Louisville, Kentucky, on the Ohio River. Taylor grew up in a small woodland cabin until, with increased prosperity, his family moved to a brick house. As a child, he lived in a battleground of the American Indian Wars, later claiming that he had seen Native Americans abduct and scalp his classmates while they were walking down the road together.[9] Louisville's rapid growth was a boon for Taylor's father, who by the start of the 19th century had acquired 10,000 acres (40 km2) throughout Kentucky, as well as 26 slaves to cultivate the most developed portion of his holdings. 