 Henry died in 1542 and his son , Henry Clifford , 2nd Earl of Cumberland , inherited the family estates . During the Rising of the North , in which Catholic magnates rebelled against Elizabeth I , Henry remained loyal to the Tudor dynasty despite the Cliffords being a Catholic family . He dismantled Appleby Castle to prevent it from being used against royal forces , and at the same time put Brougham at the service of the Elizabethan government , although there was no fighting at the castle . Under the second and third earls , Henry and George , the castle was still used as a residence , with the third earl being born at Brougham Castle . However , it was under George that the building began to decay and by 1592 it was deserted . George Clifford spent much time either in southern England in his role as Queen 's Champion or at Skipton . An inventory of the castle 's contents in 1595 demonstrates that the structure was a neglected , <unk> furnished place , and what little furniture there was old and in disrepair . 
