 In subsequent years , Berkhamsted then became closely associated with the Earls and Dukes of Cornwall . Henry III 's brother , Richard , became the Earl of Cornwall and inherited the castle from his mother Isabella , and it became a permanent part of the earldom . Berkhamsted was Richard 's favourite castle , partially because it was conveniently close to London . Richard had an impressive , three @-@ storey tower built onto the property in 1254 , and restored much of the rest of the castle ; the chroniclers of nearby Dunstable complained that his building works required so many carts to carry the timber that local trade in other goods was badly affected . The castle was used for the central administration of the earldom and Richard 's nine stewards would submit their annual financial reports there . Meanwhile , the town of Berkhamsted itself became rich as a result of the growing wool trade . Richard died at the castle in 1272 , and it passed to his son Edmund . 
