Automating creation of new VM and pushing code to server Automating creating of new VM and pushing code to server

Hi all,

I am writing a integration test for a client-server application. To simulate
the production server setup, and to update the code, I set up VMs on my
computer. The application runs on a Windows server, so I have to use systems
compatible with Windows. 

STEP 1: Set up VM properly:


Download the OVA file from MS
create a new VM from the OVA file
reset the mac id
set a static dns ip address and a VM ip address
open up the port 22
install bitvise ssh server
transfer public keys to the VM
install python 3.5.2
update pip
install virtualenv systemwide


STEP 2: Deploy code to VM:


create a requirements file using pip using: pip freeze -r requirements
zip the entire repo using powershell using: 
powershell.exe -nologo -noprofile -command "&amp; { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"
using pscp I transfer the files to the VM using:
pscp ./atu.zip ipython_user@10.15.33.31:/C:/Users/ipython_user/Desktop 
unzip the files again using powershell using:
powershell.exe -nologo -noprofile -command "&amp; { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"
delete the virtualenv folder called env in the folder using plink
create a fresh env folder using using virtualenv: virtualenv env
using the requirements.txt document, I install the requirements again by
direcatly using the pip installed in the virtualenv above using:
/path/to/env/Scripts/pip.exe install -r /path/to/requirements.txt
run the file with the python distribution installed in the virtualenv
using:
/path/to/python file_name.py


Because I was trying to execute relatively complex commands, which in turn had
multiple quotes (double and single), spaces etc, I had to spend quite a bit of
time just debbuging how to run these commands over the network.  I have
automated most of the deployment part(STEP 2) using a combination of
subprocess.call, plink, and pscp. However, I haven't yet automated
setting up the VM (STEP 1).

I am guessing, the above set of actions are not just something I am doing, and
probably others have done this as well. 

Currently, I am using low level tools to acheive this. While this works for
setting up Windows VM, it wont probably work for Linux based VMs or if I change
python versions use a different language etc etc.

I have also searched a bit and came across this book:
https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html

where the author goes into how to deploy to a linux server. However, it seems
that he is essentially doing what I am doing by directly running commands on
the linux server, just through the fabric wrapper. Because ssh doesn't work
as well on Windows, I am guessing, I will have to get back to the nitty-grittys
of deployment with fabric as well.

Question:


Is there a better way to setup the VM, and deploy code to the VMs, while
abstracting away the difference between different operating systems, python
versions, tools etc?
I have also heard of a git based workflow. Is that better than directly pushing the code to the server? Why so?
Instead of using custom scripts which it seems that you are using, I would consider using open-source tools, similar to what Tensibai has suggested in his comments.

I would further split your two Tasks a bit more differently, such that Step 1 would be to just "Set up the VM", and Step 2 would be to "Provision the VM with required programs and configuration".

My 2-cents would be to use tools such as Packer and Ansible.

You would use (for example), the VirtualBox Packer builder, to stand up your VM, which would handle points 1-4 of your Step 1, and then use the Ansible-local provisioner, or something to this effect, to run a playbook, which would handle the rest of Step 1, and all of Step 2.



The above suggestion would make your life a lot easier, and would be able to abstract different OS's as well as you'd just write a different Packer template and Ansible task. With installing different Python versions / Tools, you'll just need to update the Ansible role / have several different roles with the specific tools.

There are of course, other tools such as Vagrant, Puppet, and Chef, which can handle what you'd like to achieve, so it's up to you find your preferred tool.



Mini breakdown of Ansible modules you can use:


Open up port: win_firewal_rule_module
Installing bitvise and python: win_chocolatey
Setting up pip and virtual envs: pip
Transferring public keys / requirements.txt etc: win_copy


Etc. The full list of Ansible Windows Modules can be found here.