Palmer was born in Boston, Massachusetts on March 9, 1842 to Julius Auboyneau and Lucy Manning Peabody Palmer.
In 1864, he graduated at Harvard College with a bachelor's degree and three years later with a master's degree.