 An important moment in Herm 's political history was in 933 , when the Channel Islands were annexed to the Duchy of Normandy , they remained so until the division of Normandy in 1204 , when they became a Crown Dependency . In 1111 Brother Claude Panton was a hermit in " Erm " and in 1117 the then hermit , Brother Francis Franche Montague is recorded as living on " Erm " . After the annexation , Herm gradually lost its monastic inhabitants , and between 1570 and 1737 the governors of Guernsey used it as a hunting ground ; visiting to shoot , hunt , and fish . 
