Cadbury Castle is located north east of Yeovil. It stands on the summit of Cadbury Hill, a limestone hill situated on the southern edge of the Somerset Levels, with flat lowland to the north. The summit is above sea-level on lias stone.