 The marriage was not a success . By the 1840s , Natalie Benjamin was living in Paris with the couple 's only child , Ninette , whom she raised as a Catholic . Benjamin would visit them annually . While a senator , in the late 1850s he persuaded Natalie to rejoin him and expensively furnished a home in Washington for all three to live in . Natalie and their daughter soon embarked again for France . Benjamin , publicly humiliated by his failure to keep Natalie , consigned the household goods to auction . There were rumors , never substantiated , both that Benjamin was impotent , and that Natalie was unfaithful . 
