Russell was born into slavery in 1817 Lexington, Kentucky, as the mixed-race, very white son of Amelie "Milly" Crawford, a mixed-race woman described as octoroon (meaning she was 7/8 European in ancestry).
After Irvine's death in 1822, Alfred Russell and his mother were sold to Irvine's daughter Mary Owen Todd Russell Wickliffe and her husband Robert.