St Mary's Casemated Barracks were built during the Peninsular War and initially held French prisoners of war.
The Naval Barracks (later HMS Pembroke) opened in 1902; prior to this, most Naval (as opposed to Dockyard) personnel were accommodated on board their ships or on hulks moored nearby.