Palmer was born in Great Torrington, Devonshire, England, the fourth son of the Rev. John Palmer (a nephew of Sir Joshua Reynolds), and his wife Jane, a daughter of William Johnson [REF].
On 21 November 1831 Palmer married Isabella, third daughter of Dr John Gunning, C.B., who was inspector-general of hospitals at the time [REF].