 In its earliest form , the castle consisted of a stone keep , with an enclosure protected by an earthen bank and a wooden palisade . When the castle was built , Robert de Vieuxpont was one of only a few lords loyal to the king in the region . The <unk> were a powerful land @-@ owning family in North West England and also owned the castles of Appleby and Brough . In 1264 Robert de Vieuxpont 's grandson , also named Robert , was declared a traitor and his property was confiscated by Henry III . Brougham Castle and the other estates were eventually returned to the Vieuxpont family , and stayed in their possession until 1269 when the estates passed to the Clifford family through marriage . 
