Restricting access to a group of contacts
No, ACLs are about including groups you can see, not about excluding the ones you can't.

What we did for a case like you was to automatically add any new contact into a "visible by everyone" group and remove it if/when they are added to the "VIP" group. So what you do with smartgroups, but because we add/remove directly, it was faster as no need to wait for the cache to rebuild. It was a very long time ago, so probably nothing I can share anymore, but with two hooks in a small extension, it should be possible

Have a look at civirules, it might even be possible to do it without code.