Physics simulations take way too long to bake
Separate your rendered world from your physical world.

With soft body physics, your vertex count is the biggest thing that's going to impact the time it takes to calculate.  Physics complexity is combinatorial in nature.  Each bit capable of interacting has to check with every other bit capable of interacting.  So doubling your vert count slows down your scene by much more than half.

How do you get around this?  By using simplified proxy physics objects.  You may need a thousand verts to render your beautiful sphere with nice glossy reflections, but do you need a thousand verts to simulate its physics?  No.

Look at your first image.  In that image, you have wireframes of four objects, each of which has more than a hundred vertices.  Yet the actual profile of these objects is pretty flat.  Do you need all those vertices for physics, or do you only need those vertices for rendering?  If you don't need them for physics, make an octagon mesh collider instead.  Simply disable rendering on your new octagon collider, and possibly parent your rendered object to the octagon, rather than involving these thousand vertices in your simulation.

What if you have something soft and squishy, and the render needs to represent that?  One way to handle this is to use a physics object as the target of a mesh deform for your rendered object.  Another is to use an armature with constraints that track vertex groups on your physics object.  These kinds of deformations can be computed much more quickly than physical deformations can.  Use your physics as a guide for your render.  Don't try to render physics directly.