The early settlement was often subject to attack from sea and land, including periodic assaults from Spain and France (both of whom contested England's claims to the region), and pirates. These were combined with raids by Native Americans, who tried to protect themselves from so-called European "settlers," who in turn wanted to expand the settlement. The heart of the city was fortified according to a 1704 plan by Governor Johnson. Except those fronting Cooper River, the walls were largely removed during the 1720s.