1. A method for managing a shared file system, comprising: performing directory traversal of all directory paths and files in a plurality of shared directories in the shared file system by multiple processing nodes in the shared file system traversing the shared directories in parallel; performing a plurality of parallel execution threads in each processing node, utilizing local mutual exclusions to resolve contention for access to a plurality of local process resources; maintaining a local workload queue of elements on each processing node, the elements representing a plurality of un-read shared directories; reporting workload queue length by each processing node to a master node; monitoring a processing workload and workload queue length of the multiple processing nodes performing the directory traversal by the master node; and re-balancing the processing workload across the multiple processing nodes based on the monitored workload queue length at each processing node by the master node by determining whether each processing node is under-utilized or over-utilized by comparing the monitored workload queue length with a predetermined queue length threshold, and redistributing work elements from over-utilized processing nodes to under-utilized processing nodes.