Provision and configure Azure VM in private subnet (file, remote-exec, cloud-init, etc.) (Originally posted to discuss.hashicorp.com)
I’m setting up a basic multi-tier infrastructure composed of two Azure VMs:

Web server (VM1) with public IP and private IP (subnet1)
Data server (VM2) in private IP only (subnet2)

Both VM-s are based on the vanilla Ubuntu Linux images available on Azure.
Obviously, it’s very easy to provision (apt-get, Bash scripting, etc.) and configure the Web server using the file or remote-exec provisioners. I have also experimented with the clout-init for that purpose.
Questions:

What is the recommended method to provision and the data server?
Is cloud-init the only option?

AFAIU, the Terraform provisioners running via SSH are not an option because the data server (VM2) is not available at public IP.
What I need to do on the VM2 is:

install some packages
upload/edit some configuration files
download some data files
start some services

I’m running Terraform on Windows host, so I did not try the Ansible yet, but AFAIU it operates via SSH, similarly to the Terraform file or remote-exec provisioners.
Hashicorp recommends using provisioners only as a last resort. They recommend using the Azure VM custom_data method, or cloud-init where available.
Ansible would likely be overkill unless you will need ongoing in-place configuration management. But there are ways to make VM2 securely accessible to Ansible or provsioners if necessary.