title: convert df.apply to spark to run parallely iusing all the cores
body: <p>We have a <a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html" rel="nofollow noreferrer">panda dataframe</a> that are using.  We have a function we use in retail data which runs on a daily basis row by row to calculate the item to item difference like below</p>
  <pre><code>for itemiter in range(len(RetDf)):
      column = RetDf.loc[itemiter , &quot;username&quot;]
      RetDf[column] = RetDf.apply(lambda row: ItemDiff(RetDf.loc[itemiter, 'Val'], row['Val']), axis=1)
  </code></pre>
  <p>is there a way to convert it to sparkcontext rdd parallelize to use all cores</p>
  <pre><code>Sample Data with dummy values for retdf: 
  
  username    UserId     Val
  abc75757    1234       [0.0 , 0.0, 1.0, 2.0] 
  abcraju     4567       [0.0 , 0.0, 1.0, 2.0]
  xyzuser     4343       [0.0 , 0.0, 1.0, 2.0]
  user4abc    2323       [0.0 , 0.0, 1.0, 2.0]
  
  
  FinalOutput: 
  
  username    UserId     Val                     abc75757  abcraju        xyzuser     user4abc
  abc75757    1234       [0.0 , 0.0, 1.0, 2.0]   2.0       0.0            0.0         1.0
  abcraju     4567       [0.0 , 0.0, 1.0, 2.0]   2.0       0.0            0.0         1.0
  xyzuser     4343       [0.0 , 0.0, 1.0, 2.0]   2.0       0.0            0.0         1.0
  user4abc    2323       [0.0 , 0.0, 1.0, 2.0]   2.0       0.0            4.0         1.0
  
  ItemDiff
  
  def ItemDiff(z1,z2):
      distance_t = 0.0
      path_t = [(0,0)]
      distance_t, path_t = fastdtw(z1,z2)
      return(distance_t)
  </code></pre>