 Another theory is that it was destroyed by the Scots in May 1318 during Middleton's imprisonment in the Tower of London.It was certainly destroyed by 1323 as records of an inquest held that year after the death of Sir Aymer de Valence state Mitford Castle to be " entirely destroyed and burnt.