 Lady Anne Clifford died at Brougham Castle in 1676 and her grandson , Nicholas Tufton , 3rd Earl of Thanet , inherited the Clifford estates . He died in 1679 , and over the next five years possession passed through his three younger brothers . Under the youngest , Thomas Tufton , 6th Earl of Thanet , Brougham Castle suffered particular neglect . In 1714 , he decided that Appleby Castle was a sufficient residence and sold the contents of Brougham Castle for £ 570 . Only the Tower of League was left untouched , but in 1723 its contents were also sold , for £ 40 By the 1750s , the castle 's only practical use was as a ready source of building material for the village of Brougham , which prospered due to investment from the Earl of Thanet . In 1794 , a record of the dilapidated state of the castle noted that " much of the interior walls have lately been removed , also , for the purposes of building houses for the adjoining <unk> " . 
