 In the late 15th century , the castle became increasingly unfashionable and was fell into decline . By the mid @-@ 16th century , it was in ruins and unsuitable for royal use . Stone was taken from the castle to build houses and other buildings in the town . The castle was almost destroyed during the construction of the London and Birmingham Railway in the 1830s . As a result , it became the first building in Britain to receive statutory protection from Parliament . In 1930 , the castle passed from the Duchy of Cornwall to the government 's control . It is maintained as a tourist attraction by English Heritage . 
