An ancient manor house on the site belonged to the Pagnell family of Newport Pagnell, but was given by them to the church.
The house then descended in the Chester family to the time of the English Civil War, when it was shelled by Parliamentary forces and eventually demolished.