He was born on June 6, 1874 to African-American parents liberated during the Civil War whose number included his future stepfather 1st Sgt.
During his elementary schooling, Gregory attended the first public school that was open to both African Americans and whites in Charleston, South Carolina.