 When George Clifford died in 1605 , his wife Margaret became dowager countess and began repairing Brougham Castle , which became her favoured residence . Margaret contended with claims to the ownership of the family estates from her brother @-@ in @-@ law Francis Clifford , 4th Earl of Cumberland , but held onto Brougham Castle . Her daughter , Lady Anne Clifford continued the restoration of the castle and other Clifford properties . The only one of Margaret 's three children to survive childhood , Anne inherited the Clifford estates after her mother died in 1616 . The inheritance was not without incident . The Earl of Cumberland again asserted his claim to the Clifford estates , however the privy council found in favour of Anne . The solution was only temporary , and in April 1617 the king decided that the Earl of Cumberland was the rightful heir , and the Clifford estates passed to Francis Clifford . Later the same year , James I visited Scotland and on his return journey he stayed at the castles of Carlisle , Brougham , and Appleby , where expensive banquets were given in his honour . It is estimated that the festivities cost around £ 1 @,@ 200 . After this , Brougham was almost forgotten by its owner and neglected . 
