Noise issue when baking ambient occlusion to texture on spherical UV map
I figured it out. This effect occurs because these areas are "overhanging" from the point of view of a spherical projection. Pixels that would be on dark faces from inside the hole and light faces from the surface are all getting mixed up together because they share the same space on the UV map.

So, the solution is to choose a UV mapping where the projections of the faces do not overlap.