title: Kubernetes Vsphere Provisioner: Failed to provision volume with StorageClass "fast": Cloud provider not initialized properly
body: <p><strong>Summary</strong>
  I have provisioned kubernetes on several vcenter vms using kubespray and everything there is working perfectly.  However I am running into trouble with setting up Vsphere as the cloud provider and dynamically provisioning persistent volumes. I have tried following this <a href="https://github.com/kubernetes-sigs/kubespray/blob/master/docs/vsphere.md" rel="nofollow noreferrer">guide</a> on setting up Vsphere as a provider. All of the pods seem to launch:</p>
  <pre><code>vsphere-cloud-controller-manager-5h8vr        1/1     Running   0          77m
  vsphere-csi-controller-0                      5/5     Running   0          52m
  vsphere-csi-node-9gcn5                        3/3     Running   0          23m
  vsphere-csi-node-lcmvn                        3/3     Running   0          23m
  vsphere-csi-node-pgbcj                        3/3     Running   0          23m
  vsphere-csi-node-rvx4m                        3/3     Running   0          23m
  </code></pre>
  <p>However if I describe the pvc I get this error:</p>
  <pre><code>  Warning  ProvisioningFailed  64s (x30 over 52m)  persistentvolume-controller  Failed to provision volume with StorageClass &quot;fast&quot;: Cloud provider not initialized properly
  </code></pre>
  <p>The only error I can find is from the vsphere-csi-nodes:</p>
  <pre><code>k logs vsphere-csi-node-pgbcj -c vsphere-csi-node
  
  I0709 18:03:55.515251       1 service.go:88] configured: csi.vsphere.vmware.com with map[mode:node]
  time=&quot;2020-07-09T18:03:55Z&quot; level=info msg=&quot;identity service registered&quot;
  time=&quot;2020-07-09T18:03:55Z&quot; level=info msg=&quot;node service registered&quot;
  time=&quot;2020-07-09T18:03:55Z&quot; level=info msg=serving endpoint=&quot;unix:///csi/csi.sock&quot;
  I0709 18:03:57.645866       1 config.go:261] GetCnsconfig called with cfgPath: /etc/cloud/csi-vsphere.conf
  I0709 18:03:57.645925       1 config.go:265] Could not stat /etc/cloud/csi-vsphere.conf, reading config params from env
  E0709 18:03:57.645968       1 config.go:202] No Virtual Center hosts defined
  E0709 18:03:57.645983       1 config.go:269] Failed to get config params from env. Err: No Virtual Center hosts defined
  I0709 18:03:57.645995       1 node.go:471] Config file not provided to node daemonset. Assuming non-topology aware cluster.
  </code></pre>
  <p>Looking through &quot;secrets&quot; it looks as though all of that information was propagated correctly.</p>
  <p><strong>Pre-reqs in place</strong></p>
  <ol>
  <li>VM hardware-version &gt;15</li>
  <li>UUID enabled on all vms</li>
  <li>Vsphere &gt;6.7</li>
  </ol>
  <p><strong>Environment</strong></p>
  <ol>
  <li>Single Master</li>
  <li>Three Nodes</li>
  <li>Host OS: Debian 10.04 64bit</li>
  <li>Kubespray: latest</li>
  <li>Kubernetes: 1.18.5</li>
  <li>VM &quot;hardware version&quot;: 17</li>
  <li>Esxi 7</li>
  <li>Vcenter 7</li>
  <li>CNI Weave</li>
  </ol>
  <p><strong>Config Files</strong></p>
  <p>group_vars/all.yml</p>
  <pre><code>cloud_provider: &quot;external&quot;
  
  external_cloud_provider: &quot;vsphere&quot;
  </code></pre>
  <p>vsphere.conf</p>
  <pre><code># Vsphere provider
  external_vsphere_vcenter_ip: &quot;serveriphere&quot;
  external_vsphere_vcenter_port: &quot;443&quot;
  external_vsphere_insecure: &quot;true&quot;
  external_vsphere_user: &quot;localadminaccout@vsphere.local&quot;
  external_vsphere_password: &quot;indestructiblepasswordhere&quot;
  external_vsphere_datacenter: &quot;LAB&quot;
  external_vsphere_kubernetes_cluster_id: &quot;k8&quot;
  
  vsphere_cloud_controller_image_tag: &quot;latest&quot;
  vsphere_syncer_image_tag: &quot;v1.0.2&quot;
  vsphere_csi_attacher_image_tag: &quot;v1.1.1&quot;
  vsphere_csi_controller: &quot;v1.0.2&quot;
  vsphere_csi_liveness_prove_image_tag: &quot;v1.1.0&quot;
  vsphere_csi_provisioner_image_tag: &quot;v1.2.2&quot;
  vsphere_csi_node_driver_registrar_image_tag: &quot;v1.1.0&quot;
  vsphere_csi_driver_image_tag: &quot;v1.0.2&quot;
  
  vsphere_csi_controller_replicas: 1
  
  
  vsphere_csi_enabled: true
  </code></pre>
  <p>Storage Class</p>
  <pre><code>---
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: fast
  provisioner: kubernetes.io/vsphere-volume
  parameters:
    datastore: LAB
    diskformat: thin
    fsttype: ext4
  </code></pre>
  <p>Persistent Volume Claim</p>
  <pre><code>---
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: test
    namespace: kube-system
  spec:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 2Gi
    storageClassName: fast
  </code></pre>