title: Parallel Processing of Loop of Pandas Columns
body: <p>I have the following code which I would like to speed up.</p>
  <p>EDIT: we would like the columns in 'colsi' to be shifted by the group columns in 'colsj'. Pandas allows us to shift multiple columns at once through vectorization of 'colsi'. I loop through each group column and perform the vectorized shifts. Then I fill the NAs by the medians of the columns in 'colsi'. The reindex is just to create new blank columns before they are assigned. The issue is that I have many groups and looping through each is becoming time consuming.</p>
  <p>EDIT2: My goal is to engineer new columns by the lag of each group. I have many group columns and many columns to be shifted. 'colsi' contains the columns to be shifted. 'colsj' contains the group columns. I am able to vectorize 'colsi', but looping through each group column in 'colsj' is still time consuming.</p>
  <pre><code>colsi = ['a', 'b', 'c']
  colsj = ['d', 'e', 'f']
  med = df[colsi].median()
  
  for j in colsj:
      newcols=[j+i+'_n' for i in colsi]
      newmed = med.copy()
      newmed.index=newcols
      df = df.reindex(columns=df.columns.tolist()+newcols) 
      df[newcols] = df.groupby(j)[colsi].shift()
      df[newcols] = df[newcols].fillna(newmed)
  </code></pre>
  <p>Parallelization seems to be a good way to do it. Leaning on <a href="https://stackoverflow.com/questions/59829286/make-for-loop-execute-parallely-with-pandas-columns">this code</a>, I attempted the following but it didn't work:</p>
  <pre><code>from multiprocessing.pool import ThreadPool
  pool = ThreadPool(processes=3)
  
  colsi = ['a', 'b', 'c']
  colsj = ['d', 'e', 'f']
  med = df[colsi].median()
  
  def funct(j):
      newcols=[j+i+'_n' for i in colsi]
      newmed = med.copy()
      newmed.index=newcols
      df = df.reindex(columns=df.columns.tolist()+newcols) 
      df[newcols] = df.groupby(j)[colsi].shift()
      df[newcols] = df[newcols].fillna(newmed)
  
  for j in colsj:
      pool.apply_async(funct, (j))
  </code></pre>
  <p>I do not have any knowledge on how to go about parallel processing, so I am not sure whats missing here. Please advise.</p>