How to smoothen out a pit in a curved surface TL;DR: What is the best way to make the selected part in the image an as smooth as possible extension of the surrounding curved surface?

I've been searching extensively the past hour, but cannot find the Q&amp;A that I am looking for. I am trying to smoothen out pits in a curved surface. This is my mesh:



I have been asked to create a snap fitting for an airpod from this 3D scan. To do so, I am gonna subtract the shape of the airpod out of a bigger block. But to to that, I need a smooth surface to start with. I can not make a snap fit from an airpod with pits (you wouldn't be able to press the airpod in the snap fit).

I tried doing a limited dissolve, after which I deleted all the surfaces that were part of the pit. The hole that is created this way I tried to fill with the Fill and Grid fill functions, but this either gave non smooth or irregular surfaces.
One way is to use the Cast modifier.


Select the area you want to 'smoothen' and add it to a vertex group (I named it ball part)
Add a Cast modifier set to Sphere and limited to the vertex group.
Use the Factor, Radius and Size sliders to correct the shape.