 Ellsworth's twin sons followed their father into public service.William Wolcott Ellsworth married a daughter of lexicographer Noah Webster and became Governor of the State of Connecticut, a United States Congressman and a justice of the Connecticut Supreme Court.His twin brother, Henry Leavitt Ellsworth, served as mayor of Hartford, then was appointed the first commissioner of the U.S. Patent Office.