How to remove the default cache tags on a block?
Nope, you can't really opt out of those.

They are defined as required cache contexts in services.yml. So even if you'd manage to remove them, they are added in Renderer.php.

You could change that globally, but you have to be aware of the risks as it is likely that not every contrib/custom code is adding the user.permissions cache context when required, so we decided to make that a default so that at least that kind of cache/acess separation is enforced.