title: How to force spark worker to run a python function
body: <p>I want to run some data in Databricks (i.e. I'm using a spark cluster) through a C binary, but I'd like to do it as quickly as possible by having each worker node take on a part of the load. To do that, I have to distribute the binary to each of the nodes, which I'm attempting to do like so:</p>
  <pre class="lang-py prettyprint-override"><code>import shutil
  
  def copyFile(filepath):
      shutil.copyfile(f'/dbfs{filepath}', filepath)
      os.system(f'chmod u+x {filepath}')
  
  # The cluster has 16 workers
  sc.parallelize(range(16)).repartition(16).map(lambda s: copyFile(&quot;/tmp/fast_func&quot;))
  </code></pre>
  <p>My attempt there is to create exactly 16 partitions in the hopes that each of the 16 workers is therefore running the copyFile() function a single time (the <code>/dbfs</code> directory is shared between them all, but I don't seem to be able to set execute permissions there). I am then trying to run the binary like so:</p>
  <pre class="lang-py prettyprint-override"><code>output = inputDf.rdd.repartition(16).map(lambda s: f&quot;{s.SearchString},{s.DisplayName}&quot;).pipe(&quot;/tmp/fast_func&quot;).collect()
  </code></pre>
  <p>The inputDf is two columns named SearchString and DisplayName, which I'm concatenating into a single string and piping into the binary (which reads from stdin). Broadly speaking this works, but sometimes instead results in an error saying that the /tmp/fast_func file cannot be found, from which I assume that my attempt to force copyFile() to be run on each worker must not be correct.</p>
  <p>How can I force this to work properly?</p>