Deploying to VSphere with Packer and/or Terraform? Scenario: 

Creating multiple VMs and deploying them to VSphere. Current development uses Packer and Ansible to provision a Fusion VM, with the aim of using Terraform to deploy to VSphere.

Issue:

I've been having loads of issues with uploading Fusion VMs to VSphere and having to use work arounds such as this, and been wondering if there is a better of doing this.

Question: 

I'm questioning the use of Packer in this scenario (which deploys to Fusion), and doesn't seem to currently support a Packer->ESXi (VSphere) builder unless it's an ISO.

Would it be possible to remove Packer entirely from this scenario, so instead of building a Fusion image, provisioning the image, and uploading it to VSphere, use Terraform to deploy an ESXi image (template), and provision that image in-place using Ansible as a provisioner? (such that everything is done in VSphere rather than Fusion then VSphere).

What are the advantages and disadvantages of doing this?
I'm using vagrant with vsphere plugin, starting from a 'base' image already present on vsphere.

The new machine is then fully configured (inlcluding middleware) by chef after vsphere customization. 

That's one in many possibilities, unsure this question lead to actionnable answers out of everyone 'workaround'.