The castle was built by reforming churchman John Carswell, who was Rector of Kilmartin, Chancellor of the Chapel Royal at Stirling, and later titular Bishop of the Isles.
On Carswell's death in 1572, the castle passed to his patron, the Earl of Argyll.