Bare-Metal Kubernetes storage in enterprise Problem in a nutshell

What kind of storage for dynamic provisioning in a bare-metal kubernetes should i use.

Our configuration


Each of our kubernetes nodes (worker and master) are running on VMs. The VMs themselves run on a VCenter on top of an Hyperconverged architecture
So far the cluster have no persistent storage configured (other than the host storage)
We do have an HDFS (Hadoop Distributed File System) running in bare metal as well, to make it short, hdfs is a distributed and Highly Available storage.


What do i want

I've seen Ceph&amp;Rook being a popular solution. But if i understand it properly it required each kubernetes nodes (if i want it to be fault resistant) to have an attached physical storage, which will all contains more or less the same datas.

What's the problem with that

If each nodes have 1Tb of storage, i will have 1Tb of storage as a ceph&amp;rook storage, and any time i want to add a node to the cluster i will be forced to add 1To of storage to the new node as well.

Ceph will duplicate data across the cluster to make it available in case of failure. But since the cluster is running on top of a hyperconverged architecture, the cepth duplication will be duplicated again by the hyperconverged architecture... you may start to see the problem.

The above example makes no sense to me, especially when i have an HDFS next to this cluster.

My question is what is my best option to use the HDFS as a dynamic storage provisioner?
A storage solution for kubernetes is a pain for those of us who chose to run kubernetes on premises.

What storage solution one picks would certainly depend on potential usage, a solution that will perform well for simple file operations would not pan out very well if you want to run a database in kubernetes.

Given your configuration, I think "vSphere Cloud Provider" is your best option


  VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.


architecture overview of vSphere Cloud Provider

If you don't want to run databases you can use NFS using HDFS setup you have or any other NFS server.

Disclaimer: I have not used vSphere Cloud Provider or ceph myself. Personally, I have used NFS from our storage server for production and it works fine for moderate database loads. We also have a large gluster implementation supporting close to thousand volumes, however, based on issues we have seen with gluster I will not suggest that to anyone.

On a side note, you don't "have" to use all of your cluster nodes as "storage nodes" for Ceph &amp; Rook. Only storage nodes will have restrictions on them. If you are going with Ceph &amp; Rook, I will suggest to use dedicated storage nodes otherwise it may impact performance of your workloads.