Where do you start with hosting the infrastructure for your infrastructure?
Not a stupid question, but it's lacking details. If by stack you mean Terraform, Vault, Consul etc, then you would go about provisioning machines/VMs/containers for those services likely using some form of ansible or equivalent.

It would be prudent to store those scripts in git or any other source control system.


  Like, for example, should I just be spinning up individual VMs to host each part of the HashiCorp stack? Should I be doing that instead using containers?


Personally, I find containers more flexible especially when upgrading/ testing. The only downside is you have to remember to mount a data volume into the container to persist any configuration data which would normally live beyond the lifecycle of the container.