Payne was born in Richmond, Virginia, in 1819 to former slaves who were each of mixed race, of European and African ancestry.
Aside from religion, the young Payne showed interest in politics and economics.