1. A method for managing a shared file system, comprising: performing a parallel directory traversal of all directory paths and files in the shared file system including multiple processing nodes, utilizing a node-wise parallel directory traversal; maintaining a local workload queue of elements on each node, the elements representing a plurality of un-read directories; and monitoring a traversal processing workload of a plurality of nodes and dynamically re-balancing the workload across the plurality of nodes based on the monitored workloads by a master node