Magee was purported to have been born in North Carolina in 1841 to slaves Ephraim and Jeanette, who were held and worked on the J.J. Shanks plantation.
Magee was quoted as saying, "I was 22 years old, and all I had ever known was plowing, scraping and picking cotton, sawing logs and doing other things on a farm.