 Richard II inherited <unk> Castle in 1377 ; initially the use of it was given to his favourite , Robert de Vere and , after de Vere 's fall from power and exile in 1388 , to John Holland . Henry IV lived in the castle after he deposed Richard in 1400 , and he used the property to detain rival applicants to the throne . During this period Geoffrey Chaucer , later famous for his Canterbury Tales , oversaw renovation work on the castle in his role as a clerk . Both Henry V and Henry VI owned the castle , the latter making use of it until his overthrow in 1461 . 
