Plimmer was born at a village called in contemporary accounts "Upton-under-Amon" near Shrewsbury, Shropshire, England on 28 June 1812, youngest but one of 12 children of Isaac Plimmer, builder and timber merchant, and his wife Mary (nee Roden).
Educated at a local parish school, he was intended for teaching but preferred to train as a plasterer and master builder.