it was decided to increase the size of the garrison at the castle from 30 to 70 soldiers, due to concerns about a potential Royalist attack from the island of Jersey. Most of the soldiers lived outside the castle itself. The annual cost of this force was around £78 and in 1655 the garrison was made smaller again to reduce costs. When Charles II returned to the throne in 1660, he demobilised most of the existing army and the following year the garrison at Yarmouth was given four days notice to leave the castle. The King announced that the castle's artillery would be sent to Cowes, unless the town of Yarmouth agreed to take over the financial responsibility of running the site themselves. The town declined to do so, but Charles repeated the offer in 1666; this time Yarmouth seems to have taken action, appointing four soldiers for a garrison, although the town did not assign an officer to command them, or apparently make any repairs to the now dilapidated castle. The Crown took over the castle again in 1670, and Robert Holmes, the new Captain of the Isle of Wight, had some of the guns brought back from Cowes to the castle. The site was refortified and a new battery placed on the adjacent quay, but the older earthworks were demolished and the moat was filled in. Holmes built a mansion for himself alongside the castle, where on three occasions he hosted the King. In 1688, Charles' brother, James II, faced widespread revolt and a potential invasion of England by William of Orange. Holmes was a supporter of James, but although he intended to control Yarmouth Castle on the monarch's behalf, the local inhabitants and the garrison at Yarmouth sided with William, preventing him from openly siding with the King. 18th–21st centuries Yarmouth Castle continued to be used, and records from 1718 and 1760 show it was equipped with eight and five guns along the castle and the quay platforms, respectively. Throughout this period it was probably staffed by a captain and six gunners, supported by the local militia. In the early 18th century, Holmes' mansion was rebuilt, forming its current appearance. By the 18th century, however, Yarmouth Harbour had gradually silted up and been destroyed by industrial developments, reducing the value of the anchorage, and the design of the castle had become outdated. In 1813, during the Napoleonic