[2] He had a brother William, four years his senior, and an older sister Elizabeth, who died in childhood. Johnson's birth in a two-room shack was a political asset in the mid-19th century, and he would frequently remind voters of his humble origins.[3][4] Jacob Johnson was a poor man, as had been his father, William Johnson, but he became town constable of Raleigh before marrying and starting a family. Both Jacob and Mary were illiterate, and had worked as tavern servants, while Johnson never attended school[4] and grew up in poverty.[4] Jacob died of an apparent heart attack while ringing the town bell, shortly after rescuing three drowning men, when his son Andrew was three.[5] Polly Johnson worked as a washerwoman and became the sole support of her family. Her occupation was then looked down on, as it often took her into other homes unaccompanied. Since Andrew did not resemble either of his siblings, there are rumors that he may have been fathered by another man. Polly Johnson eventually remarried to a man named Turner Doughtry, who was as poor as she was.[6]
Johnson's mother apprenticed her son William to a tailor, James Selby. 