The hillfort is on the highest point of Mynydd-y-Gaer. It is one of three hillforts on this broad upland ridge; Craig Ty-Isaf is a much smaller hillfort to the south-west, close to Baglan, whilst Gaer Fawr is to the north. Mynydd-y-Gaer has a rounded hilltop, with grassland fields and stone walls running up to Buarth-y-Gaer. The hill stands above the Neath estuary and Briton Ferry, on the other side of Briton Ferry Woods, some to the north-west. The village of Baglan is at the foot of the hill, to the south-west. A slightly higher mountain, Foel Fynyddau lies to the east. Although the hillfort lies within Briton Ferry community, the south and east banks of the fort are on the boundary with Baglan community.