The stone castle was built at the western end of the village by John, Lord Neville in the late fourteenth century.
Upon the death of Richard Neville in 1471 at the Battle of Barnet, his lands were given to Richard, Duke of Gloucester, brother of Edward IV.