 Henry Clifford lived until 1523 . Under him and his son – also called Henry , who later became Earl of Cumberland – the castle was intermittently in use as a residence for the family . After Brough Castle was destroyed in a fire in 1521 it is likely that Brougham became the new administrative centre and focus of the local lordship . As Earl of Cumberland Henry controlled Penrith and Carlisle , although he was an unpopular landlord . When the north of England rose up in the Pilgrimage of Grace in 1536 , Henry was one of those targeted by the rebels . He confronted the rebel leaders at Kirkby Stephen in February 1537 , and after his defeat he retreated to Brougham Castle . After the Pilgrimage of Grace was suppressed , there were reforms of regional government in the north west . One of the results was that the title of Earl of Cumberland no longer gave Clifford wardenship of Penrith and Carlisle , with Brougham Castle once again becoming the Clifford 's northern @-@ most castle . 
