Python packages for Spark on datanodes
Yes, putting the packages on a NAS mount to which all the datanodes are connected will work up to dozens and perhaps 100 nodes if you have a good NAS.   However,  this solution will break down at scale as all the nodes try to import the files they need.  The Python import mechanism usese a lot of os.stat calls to the file-system and this can cause bottle-necks when all the nodes are trying to load the same code.
We want to use Python 3.x with packages like NumPy, Pandas,etc. on top of Spark.
We know the Python distribution with these packages needs to be present/distributed on all the datanodes for Spark to use these packages.
Instead of setting up this Python distro on all the datanodes, will putting it on a NAS mount to which all datanodes are connected work?
Thanks