Raiford was the youngest of eight siblings born to Ernest E. and Nannie Tillery Raiford of Greensboro, North Carolina, and the brother of lawman and humanitarian Conrad L. Raiford.
Raiford was married to Edith Leone of Italy, with whom he bore a son, Kim Leone Raiford.