Chamfer or bevel an edge with a smoothly varying depth, as if whittled or gouged out
What an amazingly detailed post!
Would something like this work for you?

To save too much adjustment in separate places, starting with 1/8
square tile, open on all the sides you can't see, here...
Assign Edge Bevel weights to the edges shown in shades of blue, all at 1 will do to start with...



Assign 2 Mirror modifiers, one about the Empty rotated  in Z by 45, and another in X,Y,and Z.



Assign a Bevel modifier, by width, limited by Weight, segments about 5, with a profile of about 0.15



Assign a second Bevel modifier, to hold the sharp edges under subdivision,limited by Angle, up around 80 degrees, and a Subdivision Surface modifier.


Tweak the Edge Bevel weights, and other Bevel parameters, to get the scalloped shape you want. If you can't get all the way there, you may have to apply the first Bevel modifier, and tweak that in the geometry, by hand.
Edit: Just noticed your first Bevel should probably have only 1 segment..