 He was the brother of Eleanor (née Jay) Chapman, wife of Henry Grafton Chapman Jr. (the president of the New York Stock Exchange, son of abolitionist Maria Weston Chapman, and father of John Jay Chapman); John S. Jay, who died young; Augusta (née Jay) Robinson, wife of Edmund Randolph Robinson (son of prominent civil engineer Moncure Robinson); and Mary (née Jay) Schieffelin, wife of William Henry Schieffelin (son of author Samuel Schieffelin).His paternal grandparents were Judge William Jay and Augusta (née McVickar) Jay.He was the great-grandson of Founding Father John Jay, the 1st Chief Justice of the United States.