pilgrims had been travelling here for years , or so the locals said , though not the type of worshippers that the current church would have approved of , for it was rumoured to have been the site of a pagan temple .