 Pollard was born at Horsham, Sussex, on 10 June 1828, the ninth child of James Pollard (1789–1851) and his wife, Susannah.He became a junior teacher at the Friends' School, Croydon in 1843, and in 1849 entered the Flounders Institute at Ackworth, Yorkshire, a Quaker college for training schoolmasters.He was appointed a master at the Quaker Ackworth School in 1851 and remained there for 16 years.