How can I eliminate the need for an Ansible control machine?
The solution is to automate setting up the control machine, i.e., the VM. You can use, for example, Vagrant, Terraform or similar tools (I'll stick with Vagrant in this answer for this example, it's about the principle). With that approach, the VM is wholly defined by a text file ("Vagrantfile"), and can easily and repeatably be created and re-created. The Vagrantfile species a base image, for example some particular Linux distribution; networking etc.; and necessary command lines to set it up, going from there.

You can set up your VM (i.e., install Ansible) by just providing all necessary shell commands. In this particular example, this is probably enough. If you need a more complex VM, you can get kind of recursive, and configure the VM itself by using Ansible (or one of its alternatives).

Obviously, you will want to treat the Vagrantfile the same as your other configurations; i.e., commit it to your SCM and keep it up to date.

This is just an example: You don't need to use Vagrant or a VM; you can also install your Ansible inside a Docker image or whatever your cloud provider of choice provides as containerization.