Bare-Metal Kubernetes storage in enterprise
A storage solution for kubernetes is a pain for those of us who chose to run kubernetes on premises.

What storage solution one picks would certainly depend on potential usage, a solution that will perform well for simple file operations would not pan out very well if you want to run a database in kubernetes.

Given your configuration, I think "vSphere Cloud Provider" is your best option


  VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.


architecture overview of vSphere Cloud Provider

If you don't want to run databases you can use NFS using HDFS setup you have or any other NFS server.

Disclaimer: I have not used vSphere Cloud Provider or ceph myself. Personally, I have used NFS from our storage server for production and it works fine for moderate database loads. We also have a large gluster implementation supporting close to thousand volumes, however, based on issues we have seen with gluster I will not suggest that to anyone.

On a side note, you don't "have" to use all of your cluster nodes as "storage nodes" for Ceph &amp; Rook. Only storage nodes will have restrictions on them. If you are going with Ceph &amp; Rook, I will suggest to use dedicated storage nodes otherwise it may impact performance of your workloads.