Carter was born in Newton, Massachusetts in 1914, and made his first ascent of Mount Washington (1,917 metres) at the age of five. He graduated from Milton Academy in Milton, Massachusetts in 1932 and from Harvard College in 1936.