The son of Hugh Pollen, a dock master, Pollen was born in Ringsend, Dublin.
On 18 May 1846, Pollen married Jane Henderson, the daughter of an officer of the Royal Navy (Lieutenant Essex, RN, of Demarara).