Singer was born in 1863 in Yonkers, New York, to Isaac Singer, the founder of the Singer Sewing Machine Company, and his wife Isabella Eugénie Boyer, a French model.
His father died in 1875 and the children, with their mother, inherited substantial wealth.