In 1865, he married his relative, Harriet M. Durfee of Fall River, with whom he had seven children, including three sons; Bertram Howard, Matthew Sterling and Howard Seymour [REF].
His business career began in 1864, when he entered one of New York's leading dry goods house as a stock boy.