James Henry Carpenter was born in Brooklyn, Kings County, New York, on September 14, 1846, to Charles H. Carpenter and Elizabeth Mary Baylis.
Carpenter spent time on Long Island as a child, both in Jamaica, Queens, and in Brooklyn.