 His maternal grandparents were Eleanor and Hickson Woolman Field (uncle to Benjamin Hazard Field).His grandfather was a prominent merchant in New York City.Jay entered Columbia College, graduating in 1859.