Gray was born in Boston, Massachusetts, to the prominent Boston Brahmin merchant family of William Gray.
Horace Gray's half-brother, John Chipman Gray went on to become a lawyer and long-time professor at Harvard Law School.