title: replace for loop to parallel process in pyspark
body: <p>I am using for loop in my script to call a function for each element of size_DF(data frame) but it is taking lot of time. I tried by removing the for loop by map but i am not getting any output.<br>
  size_DF is list of around 300 element which i am fetching from a table. </p>
  
  <p>Using For:</p>
  
  <pre><code>import call_functions
  
  newObject = call_functions.call_functions_class()
  size_RDD = sc.parallelize(size_DF) 
  
  if len(size_DF) == 0:
      print "No record present in the truncated list"
  else:
  
      for row in size_DF:
          length = row[0]
          print "length: ", length
          insertDF = newObject.full_item(sc, dataBase, length, end_date)
  </code></pre>
  
  <p>Using Map </p>
  
  <pre><code>if len(size_DF) == 0:
      print "No record present in the list"
  else:
      size_RDD.mapPartition(lambda l: newObject.full_item(sc, dataBase, len(l[0]), end_date))
  </code></pre>
  
  <p><strong>newObject.full_item(sc, dataBase, len(l[0]), end_date)</strong> 
  In full_item() -- I am doing some select ope and joining 2 tables and inserting the data into a table. </p>
  
  <p>Please help me and let me know what i am doing wrong.</p>