 Prior to the 1066 Norman conquest, the castle was held by Sir John de Mitford, whose only daughter and heiress, Sybilla Mitford, was given in marriage by William the Conqueror to the Norman knight, Richard Bertram.In the late 11th century, it was an earthwork fortress of the Bertram family, and of record as William Bertram's oppidum in 1138.In 1215, it was seized by John de Balliol, King of Scotland's troops., which was clever of him as he didn't become king until 1292.