As a bishop, Challoner usually resided in London, though on occasion, as during the "No Popery" riots of 1780, he was obliged to retire into the country.
For some years Challoner and the London Catholic priests were continually harassed in this way.