Is it possible to precompile an entire python package?
<p>We have a significant ~(50kloc) tree of packages/modules (approx 2200files) that we ship around to our cluster with each job. The jobs run for ~12 hours, so the overhead of untarring/bootrapping (i.e. resolving PYTHONPATH for each module) usually isn't a big deal. However, as the number of cores in our worker nodes have increased, we've increasingly hit the case where the scheduler will have 12 jobs land simultaneously, which will grind the poor scratch drive to a halt servicing all the requests (worse, for reasons beyond our control, each job requires a separate loopback filesystem, so there's 2 layers of indirection on the drive).</p>

<p>Is there a way to hint to the interpreter the proper location of each file (without decorating the code with paths strewn throughout (maybe overriding import?)) or bundle up all the associated .pyc files into some sort of binary blob that can just be read once?</p>

<p>Thanks!</p>
