 Following this , the Cliffords began spending more time at their other castles , particularly Skipton Castle in Yorkshire . Brougham descended through several generations of Cliffords , intermittently serving as a residence . However , by 1592 it was in a state of disrepair as George Clifford was spending more time in southern England due to his role as Queen 's Champion . The castle was briefly restored in the early 17th century to such an extent that James I was entertained there in 1617 . In 1643 , Lady Anne Clifford inherited the estates , including the castles of Brougham , Appleby and Brough , and set about restoring them . Brougham Castle was kept in good condition for a short time after Lady Anne 's death in 1676 ; however , the Earl of Thanet , who had inherited the Clifford estates , sold the furnishings in 1714 . The empty shell was left to decay as it was too costly to maintain . As a ruin , Brougham Castle inspired a painting by J. M. W. Turner and was mentioned at the start of William Wordsworth 's poem The Prelude , and was the subject of Wordsworth 's Song at the Feast of Brougham Castle upon the Restoration of Lord Clifford , the Shepherd , to the Estates and Honours of his Ancestors . The castle was left to the Ministry of Works in the 1930s and is today maintained by its successor , English Heritage . 
