Rutherfurd was born in Morrisania, New York to Robert Walter Rutherfurd and Sabina Morris, and was the grandson of U.S. Senator John Rutherfurd and great-grandson of Lewis Morris, the Signer of the Declaration of Independence.
Rutherfurd was born in Morrisania, New York to Robert Walter Rutherfurd (1788--1852) and Sabina Morris (1789--1857) of Morrisania. He was the grandson of U.S. Senator John Rutherfurd from 1791 to 1798, and great-grandson of Lewis Morris, the Signer of the Declaration of Independence.