title: Using Apache Spark to parallelize processing of a Pandas dataframe
body: <p>I have a general question regarding the appropriateness of using Spark for a type of problem I frequently encounter in Python: performing the same task on the same set of data using different parameter settings using the <code>multiprocessing</code> package.</p>
  
  <p>Consider the following toy example (note this is just an example of processing in Python; you might have used another approach):</p>
  
  <pre><code>import multiprocessing as mp
  import pandas as pd
  import numpy as np
  
  mydf = pd.DataFrame({'a':np.random.random(100)})
  
  output = mp.Queue()
  
  def count_number_of_rows_above_k(indf,k,output):
    answer = sum(indf.a &gt; k)
    output.put(answer)
  
  processes = [mp.Process(target=count_number_of_rows_above_k,args=(mydf,k,output)) for k in np.random.random(10)]
  
  for p in processes:
    p.start()
  for p in processes:
    p.join()
  
  results = [output.get() for item in processes]
  print results
  </code></pre>
  
  <p>The point is that I have a blob of data, in this case a Pandas dataframe, and I performing a standard function to it using different parameter values. I perform this in parallel and then collect the results at the end. <em>This is what I would like to do in Spark</em>, under the belief that I could scale more easily and benefit from the builtin fault tolerance. In real life, the function would of course be significantly more complex and the data would be much larger.</p>
  
  <p>In my reading on Spark, all the examples I have seen feature builtin routines using Spark dataframes. For example, counting the number of columns, summing a column, filtering, etc. I want to apply a <strong>custom</strong> function to my data.</p>
  
  <p>Is Spark appropriate for my problem? If so, how do I implement this? Do I need to push the dataframe to all the worker nodes beforehand?</p>
  
  <p>I am just asking for a few pointers. There must be documentation on this out there that I haven't found yet. Thanks.</p>