title: How do I combine multiple pandas dataframes into an HDF5 object under one key/group?
body: <p>I am parsing data from a large csv sized 800 GB. For each line of data, I save this as a pandas dataframe. </p>
  
  <pre><code>readcsvfile = csv.reader(csvfile)
  for i, line in readcsvfile:
      # parse create dictionary of key:value pairs by csv field:value, "dictionary_line"
      # save as pandas dataframe
      df = pd.DataFrame(dictionary_line, index=[i])
  </code></pre>
  
  <p>Now, I would like to save this into an HDF5 format, and query the h5 as if it was the entire csv file. </p>
  
  <pre><code>import pandas as pd
  store = pd.HDFStore("pathname/file.h5")
  
  hdf5_key = "single_key"
  
  csv_columns = ["COL1", "COL2", "COL3", "COL4",..., "COL55"]
  </code></pre>
  
  <p>My approach so far has been:</p>
  
  <pre><code>import pandas as pd
  store = pd.HDFStore("pathname/file.h5")
  
  hdf5_key = "single_key"
  
  csv_columns = ["COL1", "COL2", "COL3", "COL4",..., "COL55"]
  readcsvfile = csv.reader(csvfile)
  for i, line in readcsvfile:
      # parse create dictionary of key:value pairs by csv field:value, "dictionary_line"
      # save as pandas dataframe
      df = pd.DataFrame(dictionary_line, index=[i])
      store.append(hdf5_key, df, data_columns=csv_columns, index=False)
  </code></pre>
  
  <p>That is, I try to save each dataframe <code>df</code> into the HDF5 under one key. However, this fails:</p>
  
  <pre><code>  Attribute 'superblocksize' does not exist in node: '/hdf5_key/_i_table/index'
  </code></pre>
  
  <p>So, I could try to save everything into one pandas dataframe first, i.e. </p>
  
  <pre><code>import pandas as pd
  store = pd.HDFStore("pathname/file.h5")
  
  hdf5_key = "single_key"
  
  csv_columns = ["COL1", "COL2", "COL3", "COL4",..., "COL55"]
  readcsvfile = csv.reader(csvfile)
  total_df = pd.DataFrame()
  for i, line in readcsvfile:
      # parse create dictionary of key:value pairs by csv field:value, "dictionary_line"
      # save as pandas dataframe
      df = pd.DataFrame(dictionary_line, index=[i])
      total_df = pd.concat([total_df, df])   # creates one big CSV
  </code></pre>
  
  <p>and now store into HDF5 format</p>
  
  <pre><code>    store.append(hdf5_key, total_df, data_columns=csv_columns, index=False)
  </code></pre>
  
  <p>However, I don't think I have the RAM/storage to save all csv lines into <code>total_df</code> into HDF5 format. </p>
  
  <p>So, how do I append each "single-line" df into an HDF5 so that it ends up as one big dataframe (like the original csv)?</p>
  
  <p>EDIT: Here's a concrete example of a csv file with different data types:</p>
  
  <pre><code> order    start    end    value    
   1        1342    1357    category1
   1        1459    1489    category7
   1        1572    1601    category23
   1        1587    1599    category2
   1        1591    1639    category1
   ....
   15        792     813    category13
   15        892     913    category5
   ....
  </code></pre>