title: Alternatives to pandas.merge because of "array is too big" value error?
body: <p>I'm creating a function that inputs a dataframe (pulled from a csv file) with the format:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Date</th>
  <th>Stock1</th>
  <th>Date</th>
  <th>Stock2</th>
  <th>Date</th>
  <th>Stock3</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>01/01/2000</td>
  <td>100</td>
  <td>01/01/2000</td>
  <td>12.1</td>
  <td>01/01/2000</td>
  <td>54</td>
  </tr>
  <tr>
  <td>03/01/2000</td>
  <td>101</td>
  <td>02/01/2000</td>
  <td>12.5</td>
  <td>03/01/2000</td>
  <td>50</td>
  </tr>
  <tr>
  <td>04/01/2000</td>
  <td>104</td>
  <td>03/01/2000</td>
  <td>12.2</td>
  <td>05/01/2000</td>
  <td>49</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>for any number of stocks and any number of dates, and outputs a dataframe as follows:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Date</th>
  <th>Stock1</th>
  <th>Stock2</th>
  <th>Stock3</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>01/01/2000</td>
  <td>100</td>
  <td>12.1</td>
  <td>54</td>
  </tr>
  <tr>
  <td>03/01/2000</td>
  <td>101</td>
  <td>12.2</td>
  <td>50</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>i.e. all the dates that do not exist in all the stock price time series are discarded.</p>
  <p>The code I have so far is as follows:</p>
  <pre><code>import pandas as pd
  import numpy as np
   
  def collectdata(df):
      n = (len(df.columns))
      data = df.iloc[:,0:2].copy()
      data.columns = ['Date', data.columns[1]]
      for i in range(2,n,2):
          x = df.iloc[:,i:i+2].copy()
          x.columns=['Date', x.columns[1]]
          data = pd.merge(data, x)
      return data
   
  # example: we have a file called test.csv:
  
  dataset = collectdata(pd.read_csv('test.csv'))
  print(dataset.head())
  </code></pre>
  <p>The code works fine for smaller data sets, but when I attempted with a data set of 15 stocks with and 5000 rows I got:
  <em>ValueError: array is too big</em></p>
  <p>I believe this problem is arising because I am using pd.merge (as I can pull the dataset with no problem). Is there an alternative to this that I can use that will eliminate this error? (Or equally a more efficient way around this problem?)</p>