 Roger Clifford died at Skipton Castle in 1389 , and the Clifford family began to lose interest in Westmorland . The Cliffords preferred their properties in Yorkshire to their dilapidated castles in Westmorland , which had been ravaged by wars with Scotland . Brougham Castle is not known to have been in use as a residence again until 1421 , when a man was accused of forging coins in the castle . Although little is known about Brougham during this period , historians believe it likely that repairs were undertaken , and a rivalry emerged between the Clifford family and the House of Neville that would later have consequences for Brougham . The familial enmity meant that the Earl of Salisbury , a Neville , used his position as lord of Penrith to antagonise the Cliffords ; it is likely that Brougham Castle was kept garrisoned due to its proximity to Penrith . In the Wars of the Roses ( 1455 – 1485 ) , the two families were on opposing sides , the Cliffords supporting the House of Lancaster and the Nevilles supporting the House of York . When the Yorkist Edward IV usurped the throne in 1461 the lands of John Clifford , 9th Baron de Clifford were confiscated . In 1471 , Edward IV granted Sir William Parr Brougham Castle and other properties which had belonged to the Cliffords . A year later Henry Clifford , John 's son and heir , was pardoned and when the Lancastrian Henry Tudor overthrew the Yorkists and took the throne as Henry VII , Henry Clifford appealed for the return of the Clifford estates . This was granted in November 1485 . 
