title: How to parallelize a function the correct way
body: <p>I currently have a setup where I have a very large Pandas dataframe “df” that can be filtered into separate batches df_batch_1= df[df[‘column’]==&lt;filter 1&gt;], df_batch_2= df[df[‘column’]==&lt;filter 2&gt;],…etc. In addition, I have a very “heavy” function “heavy_function(df_batch)” that must do some heavy computation on each data frame batch df_batch_1, df_batch_2,...etc. My plan is to do this “heavy lifting” on a data bricks (pyspark) cluster since it runs too slow on a regular computer.</p>
  <p>So far, I am running this by using threads on a data bricks cluster like this:</p>
  <pre><code>threads = [threading.Thread(target= heavy_function, args=(df[df[‘column’]==filter]) for filter in [&lt;filter 1&gt;, &lt;filter 2&gt;,…etc.]]
  for t in threads:
     t.start()
  </code></pre>
  <p>I have been told that this is an anti-pattern and that I should find another way of doing this. I hope you can help pointing me in the right direction. What is the right “pysparkian” way of doing this?</p>
  <p>Any help is appreciated!</p>
  <p>Best regards,
  DK</p>