 When Edward III replaced Edward II on the throne , Robert Clifford , Roger 's younger brother , was granted most of the lands that had been confiscated . By 1333 , Robert had united under his control all the estates which had belonged to the Vieuxpont family . Hostilities between England and Scotland resumed in 1332 when Edward Balliol invaded to seize the Scottish throne for himself . He was expelled from Scotland in December 1333 . On entering Westmorland , Balliol sought refuge with the Clifford family , staying at the castles of Appleby , Brougham , Brough , and Pendragon . Robert Clifford was not heavily involved in the renewed conflict , although he did take part in battles in 1332 , 1337 , and 1342 . When the value of his property was assessed on his death in 1344 the estates of Brougham were suffering from the war , with indications that Brougham Castle was in a state of disrepair having endured the 1340s without funds for maintenance . Two minorities followed until Roger Clifford , 5th Baron Clifford , came of age in 1354 . Another truce between Scotland and England was signed in 1357 , this time lasting until 1384 . Although Roger Clifford spent much time at Appleby – which was Westmorland 's county town – he was responsible for rebuilding the domestic buildings at Brougham Castle , including the hall . He was ordered by the king to maintain a force of 40 men @-@ at @-@ arms and 50 mounted archers near the west end of the Scottish border region , and some were likely stationed at Brougham . The need for extra accommodation a possible reason why Clifford began rebuilding . In August 1388 , the Scottish launched an attack into England , with one force advancing east – and were eventually confronted at the Battle of Otterburn in Northumberland – and another raiding the west , reaching as far as Brough , 20 miles ( 32 km ) to the south @-@ east . During this time Brougham Castle was briefly captured by Scottish forces . 
