Sargent was the second son of Henrietta (Gray) and Ignatius Sargent, a Boston merchant and banker who grew wealthy on railroad investments.
He attended Harvard College, where he graduated in Biology in the class of 1862.