The wooded height of Castle Crag rises between Broadslack Gill and the Derwent, the two streams meeting to the north beneath the outlying knoll of Low Hows. It has steep faces on all sides except the south, where a low ridge runs out and then swings west around the head of Broadslack Gill. A narrow col here provides the topographic link to High Spy.