How to disable speaker independent of volume settings
I recently updated and found a similar "mute all" function missing as well, so I feel your pain :/

My recommendation is to use a very powerful app: Automate

Automate allows you to monitor various activities on your phone and then respond to them. The catch-all method to accomplish what you want would be to create a simple loop of: if volume change -> set volume zero

While that sounds easy enough, there are some pitfalls: multiple loops based on ringtone, notification, media, etc., and, depending on how your phone implements it, possible high battery drain.

So on to my actual recommendation (still using Automate): monitor for a specific app, e.g. Google Duo in your example, and automatically set volume to zero when it changes it. This is what I do.

In conjunction with that, to alleviate the issue of accidental button/toggle pressing: you can set default ringtones, notifications, etc. to "silent" (go into sound setting and pick a sound. "Silent" should be first on the list). If that's not satisfactory, Automate can also save the day. It can detect if the phone ever leaves vibrate mode and set it back to vibrate (or silent, etc.).

You've got a few options, but I highly recommend the app. It's a very powerful tool that I believe is fit for just this sort of application. Right tool for the job :) Lemme know if you need any clarifications or extra help getting it set up, and best of luck!