WASHINGTON ( CNN ) -- In January , President-elect Barack Obama and his family will make history , becoming the first African-American first family to move into the White House -- a house with a history of slavery . In fact , the legacy of American presidents owning slaves goes all the way back to George Washington . A wood engraving of handcuffed and shackled slaves passing the U.S. Capitol , depicts a scene circa 1819 . Twelve American presidents owned slaves and eight of them , starting with Washington , owned slaves while in office . Almost from the very start , slaves were a common sight in the executive mansion . A list of construction workers building the White House in 1795 includes five slaves - named Tom , Peter , Ben , Harry and Daniel -- all put to work as carpenters . Other slaves worked as masons in the government quarries , cutting the stone for early government buildings , including the White House and U.S. Capitol . According to records kept by the White House Historical Association , slaves often worked seven days a week -- even in the hot and humid Washington summers . In 1800 , John Adams was the first president to live in the White House , moving in before it was finished . Adams was a staunch opponent of slavery , and kept no slaves . Future presidents , however , did n't follow his lead . Thomas Jefferson , who succeeded Adams , wrote that slavery was an `` assemblage of horrors '' and yet he brought his slaves with him . Early presidents were expected to pay their household expenses themselves , and many who came from the so-called `` slave states '' simply brought their slaves with them . Martin Van Buren , William Henry Harrison , Andrew Johnson and Ulysses S. Grant all owned slaves but not during their time in office . James Madison , Jefferson 's successor , held slaves all of his life including while he was in office . During the war of 1812 Madison 's slaves helped remove material from the White House shortly before the British burned the building . Michelle Obama uncovers slaves in her family » . In 1865 one of Madison 's former slaves , Paul Jennings , wrote the first White House memoir : `` A Colored Man 's Reminiscences of Life in the White House . '' In the book , Jennings called Madison `` one of the best men that ever lived '' and said Madison `` never would strike a slave , although he had over one hundred ; neither would he allow an overseer to do it . '' There were other presidents who treated their slaves less kindly . James Monroe , Andrew Jackson , John Tyler , James K. Polk and Zachary Taylor all owned slaves while they were in office . The last of these , President Taylor , said owning slaves was a Constitutional right and he said slave-owners like himself would `` appeal to the sword if necessary '' to keep them . The Civil War , of course , put that opinion to the test . Now , the Obamas are moving into the White House . `` The apple cart has been turned over here when you have the Obamas -- the first African-American couple -- now actually management and you are having in some cases white Americans serving them , '' says presidential historian Doug Brinkley . Michelle Obama learned this year that one of her great-great grandfathers was a slave who worked on a rice plantation in South Carolina . She says finding that part of her past uncovered both shame and pride and what she calls the tangled history of this country . For many , the historic election on November 4 marked a new beginning . Though Michelle Obama 's ancestors had to come through the ordeal of slavery , `` Her children are sleeping in the room of presidents , '' said Brinkley . `` It 's a very great and hopeful sign . ''
Obamas will make history as first African-American first family in White House . Slaves often worked seven days a week to build White House , U.S. Capitol . Twelve presidents owned slaves ; eight owned slaves while in office . Historian : Obamas moving into White House `` a very great and hopeful sign ''