Not including Neidpath, who owned the other castle that Sir Simon Fraser had for most of its existence?
Simon Fraser (d. 1306) Sir Simon Fraser of Oliver and Neidpath was a Scottish knight who fought in the Wars of Scottish Independence, for which he was hanged, drawn, and quartered in 1306.
Fraser of Strichen The Frasers of Strichen are a line of the Clan Fraser. When the Frasers of Lovat had no heir, the Frasers of Strichen inherited the arms, name, and the Lordship of Parliament.