How to prevent a site from ever using experimental core modules?
This is by far not as black and white as you make it in regards to experimental modules.

First, on a mission critical production site, you don't want to do anything directly on production. Completely irrelevant then if it's an experimental module or not. Any configuration change or module should be tested first. There are modules to enforce a read only configuration system for example, that's one way to approach that. One example is Configuration Read-only mode, note that I have no personal experience with it.

Second, there are experimental modules and there are experimental modules. The first experimental module apart from migrate, wich is a special case anyway, was big_pipe. Which is now officially stable in 8.3. It is a module that has zero data, zero API, zero configuration, zero UI. You just enabled it and it does its thing. As such, it was perfectly fine to use it on production already and people have been doing that, myself included. After testing it first, like any other module/functionality. If you found a bug, you could disable it from one minute to the other, without loosing any data or critical functionality.

On the other side of that spectrum is content_moderation and anything around that. Once you start using it, it is a critical and central part of your content creation workflow, and getting rid of it again is much harder. We specifically made it so you could uninstall it, and your content that was published was still published and same for unpublished. But your workflows and metadata around that would be gone. Also upgrading is likely going to be tough from 8.2 to 8.3. That's also why we're working hard to get media_entity into core as a non-experimental module, so people have a stable API and data storage that they can rely on.

Third, the concept of experimental modules is pretty new and there's a bunch of modules in core that we would have made experimental if we could have back when they were added. rest.module and the related modules is one example there. 

And last, lots of contrib modules are at least as experimental as experimental core modules, they just don't warn you about that in such an obvious way. Especially development/pre-beta versions. They just don't warn you about themself in such an obvious way.

To summarize, as with many other things, you need to do your research about an experimental module before using it, and for example check if it's big_pipe-like or content_moderation-like. And then decide what that means for your site.