title: How to execute arbitrary python code on spark cluster distributed to workers
body: <p>I am trying to run a simulation in python using a spark cluster that takes the form of two steps:</p>
  <ol>
  <li><p>Execute (in parallel over a number of worker nodes) a set of python functions. The results of these are to be written out as text files</p>
  </li>
  <li><p>Collect the results. This will take place using pyspark dataframes.</p>
  </li>
  </ol>
  <p>Is it possible to instruct spark to execute python code on worker nodes in a spark cluster ( i.e. using ordinary python) to achieve this first step?
  When I try using <code>spark-submit</code>, only commands that are in spark context are executed on the spark cluster. The rest of the python code is executed on the local machine, which I do not want to do.</p>
  <p>This answer seems to say no: <a href="https://stackoverflow.com/questions/53542955/using-regular-python-code-on-a-spark-cluster">Using regular python code on a Spark cluster</a> but is not terribly specific.</p>
  <h2>Example for Clarification</h2>
  <p>To give an example of step 1, I have a script called <code>draw_from_uniform_distribution.py</code> that does as follows:</p>
  <pre><code>import numpy
  the_output_file=sys.argv[1] #get output file from command line
  the_number=numpy.random.uniform(size=1)
  f_out=open(the_output_file,'w')
  print(the_number,file=f_out)
  </code></pre>
  <p>I want to run this script 1000 times in parallel on the spark cluster. How do I do so?</p>