title: Crash when joining large dataframes with pandas on a Jupyter notebook
body: <p>I work for the first time with pandas. I am on a project for which I have yearly observations over 20 years. For each year I have around one million observations, each consisting of a 1x80 vector. Thus, I currently have 20 CSV files, one for each year, composed roughly of 1 000 000 rows by 80 columns. Each of the CSV file is around 400 000 Ko.</p>
  
  <p>I would like to join those files into a single one using pandas join function. Given that the datasets do not necessarily have overlapping observations and that the columns are labeled by year, I would end up with a several million rows by 1600 (80*20) columns CSV file.</p>
  
  <p>Now, my problem is that my computer crashes every time I try to join these files through IPython (using a jupyter notebook), probably because it is beyond its capacity to handle the join. It has 65 GB available memory and 8 GB RAM.</p>
  
  <p>The code I am using is the following:</p>
  
  <pre><code>import pandas as pd
  
  z = pd.read_csv('path_to_file/Year1.csv', low_memory=False)
  
  for year in [Year 2, Year 3, ..., Year 20]:
      y = pd.read_csv('path_to_file/'+ str(year) + '.csv', low_memory=False)
      y = set_index('observation_name')
      z = z.join(other=y, on='observation_name', how='outer')
  
  z.to_csv('path_to_file/Name.csv')
  </code></pre>
  
  <p>No error message is displayed and the code works fine when I iterate only on two or three years. But when I start iterating over more than four years, my computer just freezes quickly so that I have to shutdown. For information, after merging the three first years the dataset is already around 1 500 000 rows and 240 columns.</p>
  
  <p>Does anyone have some experience on a similar issue? How can I efficiently join these datasets? Should I use another language/a computer with more capabilities? Any advices are welcome.</p>