The castle is located in the Culm valley in the Blackdown Hills, on the west side of the village of Hemyock. The site belonged to the Hidon family in the 11th and 12th centuries, passing by marriage into the Dynham family in the 13th. In the 13th century a building was constructed on the current site, protected by a spring-fed moat.