title: Merge a large Dask dataframe with a small Pandas dataframe
body: <p>Following the example here: <a href="https://www.youtube.com/watch?v=jrc3cpf3PKU" rel="noreferrer">YouTube: Dask-Pandas Dataframe Join</a> I attempting to merge a ~70GB Dask data frame with a ~24MB that I loaded as a Pandas dataframe.</p>
  
  <p>The merge is on two columns A and B, and I did not set any as indices:</p>
  
  <pre><code>import dask.dataframe as dd
  from dask.diagnostics import ProgressBar
  
  small_df = pd.read_csv(dataframe1) # as pandas
  large_df = dd.read_csv(dataframe2) #as dask.dataframe
  
  df2 = large_df.merge(small_df, how='left', left_on=leftcolumns, right_on=rightcolumns) #do the merge
  
  A = df2[df2['some column'] == 'somevalue'] #do a reduction that would fit on my computer
  
  pbar = ProgressBar()
  pbar.register()
  
  result = A.compute()
  </code></pre>
  
  <p>I'm using a Windows computer with 16GB of RAM and 4 cores.
  I use the progress bar to assess how far along the merging process it is. I left it all night last night. I restarted it this morning and so far its about half an hour in and 0% progress.</p>
  
  <p>Thank you and I appreciate your help,</p>
  
  <p><strong>Update</strong></p>
  
  <p>I tried it on my Mac with 8GB of RAM and worked pretty well. I have the Dask distribution that comes with Anaconda I believe. I don't think I did anything different in any case.</p>
  
  <p>I share my results and time following the above coding (21 minutes):</p>
  
  <pre><code>In [26]: C = result1.compute()
  [########################################] | 100% Completed | 21min 13.4s
  [########################################] | 100% Completed | 21min 13.5s
  [########################################] | 100% Completed | 21min 13.6s
  [########################################] | 100% Completed | 21min 13.6s
  </code></pre>
  
  <p><strong>Update 2</strong></p>
  
  <p>I updated to the latest version of Dask on my Windows computer and it worked well.</p>