title: Multiprocessing a list of RDDs
body: <p>I am trying to <strong>multiprocess</strong> a <em>list</em> of RDDs as follows </p>
  
  <pre><code>from pyspark.context import SparkContext
  from multiprocessing import Pool
  
  
  
  def square(rdd_list):
      def _square(i):
          return i*i
      return rdd_list.map(_square)
  
  sc = SparkContext('local', 'Data_Split')
  data = sc.parallelize([1,2,3,4,5,6])
  
  dataCollection = [data, data, data]
  
  p = Pool(processes=2)
  result = p.map(square, dataCollection)
  print result[0].collect()
  </code></pre>
  
  <p>I am expecting a list of RDDs in output with each element containing the squared elements from <em>data</em>. </p>
  
  <p>But running the code results in the following error :</p>
  
  <blockquote>
    <p>Exception: It appears that you are attempting to broadcast an RDD or
    reference an RDD from an action or transformation. RDD transformations
    and actions can only be invoked by the driver, not inside of other
    transformations; for example, rdd1.map(lambda x: rdd2.values.coun\ t()
    * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more
    information, see SPARK-5063.</p>
  </blockquote>
  
  <p>My questions are :-</p>
  
  <p>1) Why doesn't the code work as expected ?  How can I fix this ?</p>
  
  <p>2) Will I gain any performance enhancement (in terms of lessening the runtime) for my program if I use the <strong>p.map</strong> (Pool) instead of a simple <strong>map</strong> on my RDD list.</p>