title: How to multiprocess finding closest geographic point in two pandas dataframes?
body: <p>I have a function which I'm trying to apply in parallel and within that function I call another function that I think would benefit from being executed in parallel. The goal is to take in multiple years of crop yields for each field and combine all of them into one pandas dataframe. I have the function I use for finding the closest point in each dataframe, but it is quite intensive and takes some time. I'm looking to speed it up.</p>
  
  <p>I've tried creating a pool and using map_async on the inner function. I've also tried doing the same with the loop for the outer function. The latter is the only thing I've gotten to work the way I intended it to. I can use this, but I know there has to be a way to make it faster. Check out the code below:</p>
  
  <pre class="lang-py prettyprint-override"><code>return_columns = []
  return_columns_cb = lambda x: return_columns.append(x)
  
  def getnearestpoint(gdA, gdB, retcol):
      dist = lambda point1, point2: distance.great_circle(point1, point2).feet
  
      def find_closest(point):
          distances = gdB.apply(
              lambda row: dist(point, (row["Longitude"], row["Latitude"])), axis=1
          )
          return (gdB.loc[distances.idxmin(), retcol], distances.min())
  
      append_retcol = gdA.apply(
          lambda row: find_closest((row["Longitude"], row["Latitude"])), axis=1
      )
      return append_retcol
  
  def combine_yield(field):
      #field is a list of the files for the field I'm working with
      #lots of pre-processing
  
      #dfs in this case is a list of the dataframes for the current field
      #mdf is the dataframe with the most points which I poppped from this list
  
      p = Pool()
      for i in range(0, len(dfs)):
          p.apply_async(getnearestpoint, args=(mdf, dfs[i], dfs[i].columns[-1]), callback=return_cols_cb)
      for col in return_columns:
          mdf = mdf.append(col)
  
      '''I unzip my points back to longitude and latitude here in the final 
         dataframe so I can write to csv without tuples'''
  
      mdf[["Longitude", "Latitude"]] = pd.DataFrame(
          mdf["Point"].tolist(), index=mdf.index
      )
      return mdf
  
  def multiprocess_combine_yield():
      '''do stuff to get dictionary below with each field name as key and values 
       as all the files for that field'''
      yield_by_field = {'C01': ('files...'), ...}
      #The farm I'm working on has 30 fields and below is too slow
      for k,v in yield_by_field.items():
          combine_yield(v)
  </code></pre>
  
  <p>I guess what I need help on is I envision something like using a pool to imap or apply_async on each tuple of files in the dictionary. Then within the combine_yield function when applied to that tuple of files, I want to to be able to parallel process the distance function. That function bogs the program down because it calculates the distance between every point in each of the dataframes for each year of yield. The files average around 1200 data points and then you multiply all of that by 30 fields and I need something better. Maybe the efficiency improvement lies in finding a better way to pull in the closest point. I still need something that gives me the value from gdB, and the distance though because of what I do later on when selecting which rows to use from the 'mdf' dataframe.</p>