the town had grown up around the rutherford estate , in the old medieval fashion of feudal serfs , clustering round the castle of their lord , and it was here , midway between town and great house , that the parish church was located .