 By 1269 , Roger Clifford had married Isabel Vieuxpont and possession of Brougham Castle – as well as her other property – descended through the Clifford family . In 1283 , Roger predeceased his wife , who died in 1292 . At 18 , their son Robert was not old enough to take possession of his lands . During his three @-@ year minority , his estates suffered from neglect and poaching . When the Anglo @-@ Scottish wars began in 1296 , Robert Clifford played a prominent role in the conflict . As the furthest north of his castles , Brougham became Clifford 's most important base and he spent a lot of time there . It was during this period that Clifford undertook an extensive building programme . The timber palisade surrounding the site was replaced with a stone curtain wall . A four @-@ storey stone residential tower , called the Tower of League , was built in the castle 's south @-@ west corner . A fourth storey was added to the keep , and a double gatehouse attached to its northern side . The construction of a new stone hall to the south of the keep may indicate that during the war there was a larger garrison present than in peacetime , or it may have been built in anticipation of a royal visit . In July 1300 , Edward I – himself a renowned castle builder – visited Brougham with a large household of followers and the Prince of Wales . Although it is not certain whether the king stayed at the castle , historians believe it to have been likely . In 1309 , Robert Clifford was granted a licence to crenellate Brougham Castle ; this has been taken as an indication that by this point the rebuilding was complete . Licences to crenellate granted permission for a person to fortify a site . They were also proof of a relationship with or favour from the monarch , who was the one responsible for granting permission . 
