title: Pytables efficiently read and process thousands of groups
body: <p>This should be easy enough, but it is taking me forever to compute this, and I can't think of any other ways to process my file. I'd appreciate any ideas.</p>
  
  <blockquote>
    <p><strong>TL;DR:</strong> I am looking for a way to shortcut <code>for key in store.keys()</code> and run the same analysis on all data contained within
    each node in a HDF file with 261k nodes (aka keys, groups...) such that each node is processed independent of the others.</p>
  </blockquote>
  
  <p>I have a H5 file on disk, which weighs in a couple of hundred gigabytes. The file contains hundreds of thousands of nodes (261k, to be exact), and I want to process them all using the same method, but separately. Each node (or group) contains a single table, with a date-time index and three float columns of data. I want to compute quantiles for each column in each table. Basically, here is how the H5 file would look like (partial output of <code>$ ptdump -av</code>):</p>
  
  <pre><code>/ (RootGroup) ''
      /._v_attrs (AttributeSet), 4 attributes:
       [CLASS := 'GROUP',
        PYTABLES_FORMAT_VERSION := '2.1',
        TITLE := '',
        VERSION := '1.0']
  /101P09999 (Group) '101P09999'
     /101P09999._v_attrs (AttributeSet), 15 attributes:
      [CLASS := 'GROUP',
       TITLE := '101P09999',
       VERSION := '1.0',
       data_columns := [],
       encoding := 'UTF-8',
       index_cols := [(0, 'index')],
       info := {1: {'type': 'Index', 'names': [None]}, 'index': {'index_name': 'DATETIME'}},
       levels := 1,
       metadata := {'STATE': 'Georgia', 'LENGTH': 4.86258, 'COUNTRY': 'USA', 'ROAD_NUMBER': 'US-27/GA-1', 'LATITUDE': 34.88279, 'COUNTY': 'Walker', 'LONGITUDE': -85.27023, 'ROAD_NAME': 'Lafayette Rd/Martha Berry Hwy', 'DIRECTION': 'Northbound'},
       nan_rep := 'nan',
       non_index_axes := [(1, ['TTAV', 'TTPC', 'TTFT'])],
       pandas_type := 'frame_table',
       pandas_version := '0.15.2',
       table_type := 'appendable_frame',
       values_cols := ['values_block_0']]
  /101P09999/table (Table(2345,), shuffle, blosc(5)) ''
    description := {
    "index": Int64Col(shape=(), dflt=0, pos=0),
    "values_block_0": Float32Col(shape=(3,), dflt=0.0, pos=1)}
    byteorder := 'little'
    chunkshape := (3276,)
    autoindex := True
    colindexes := {
      "index": Index(6, medium, shuffle, zlib(1)).is_csi=False}
    /101P09999/table._v_attrs (AttributeSet), 12 attributes:
     [CLASS := 'TABLE',
      FIELD_0_FILL := 0,
      FIELD_0_NAME := 'index',
      FIELD_1_FILL := 0.0,
      FIELD_1_NAME := 'values_block_0',
      NROWS := 2345,
      TITLE := '',
      VERSION := '2.7',
      index_kind := 'datetime64',
      values_block_0_dtype := 'float32',
      values_block_0_kind := ['TTAV', 'TTPC', 'TTFT'],
      values_block_0_meta := None]
  </code></pre>
  
  <blockquote>
    <p><strong>NOTE:</strong> Don't pay attention to the <code>NROWS</code> value in the above output. This output is just for February 2014, I have the data from
    all 12 months in the master table.</p>
  </blockquote>
  
  <p>Right, what I want to do is take the data in columns <code>['TTAV', 'TTPC', 'TTFT']</code>  and divide it by the group attribute <code>metadata['LENGTH']</code> (4.86258 in this case). Next I want to divide it into 6 groups based on the time stamp of the data. Within each of these 6 groups I want to then compute quantiles.</p>
  
  <p>What I have now is a novice approach:</p>
  
  <pre><code>with pd.HDFStore(store_path, 'r') as store:
      for key in store.keys()
          sens_data = store[key]
          # split_data = split the data into the required groups based on time stamp...
          for data in split_data:
              data /= store.get_storer(key).attrs.metadata['LENGTH']
              perc = split_data.quantile(q=np.arange(0.05, 1, 0.05)).transpose()
              # Create a column to contain sensor name:
              perc[0] = key[1:]
              perc.set_index(0, append=True, inplace=True)
              perc.index.rename(['DATA COL', 'SENS NAME'], inplace=True)
              # Merge perc into a dictionary of dataframes with keys the groups
              # the data was split into, and value a dataframe of appended percs
  </code></pre>
  
  <p>So, ultimately, the dictionary of dataframes will look like this:</p>
  
  <pre><code>In [1]: percentiles['night']
  Out[1]:                      0.05   0.10  ...  0.90  0.95
          DATA COL  SENS NAME
                    101P09999  115    118   ...  133   135
          TTAV      101P10000  95     100   ...  120   125
                    101P10001  108    109   ...  111   113
                       ...
                    101P09999  110    112   ...  133   135
          TTPC      101P10000  115    118   ...  133   135
                    101P10001  115    118   ...  133   135
                       ...
                    101P09999  115    118   ...  133   135
          TTFT      101P10000  115    118   ...  133   135
                    101P10001  115    118   ...  133   135
                       ...                  ...
  </code></pre>
  
  <p>(Forgive me for not entering random data for the rest of the rows. Also, though I have entered <code>int</code> values in my example DataFrame, the values will actually be <code>float32</code>, as seen in the <code>ptdump</code> output.) But this is how it would look like for all groups (night, morning, afternoon...), at the end of the above code. So basically, each of <code>['TTAV', 'TTPC', 'TTFT']</code>, there will be 261k rows, each corresponding to a sensor.</p>
  
  <p>Now clearly, <code>for key in store.keys()</code> is not the way to go. (It takes about 15 minutes to just retrieve all the keys!) I find infinite examples where data for only one group and table is retrieved and processed, but nothing to help process all groups. Any thoughts? Parallel read is OK, though I have not got it to work yet (it is throwing an UnImplimented Error, when I use key to access data within nodes in separate processes. Maybe I need to pass the store to the function as well). However, the biggest problem is with <code>for key in store.keys()</code> taking 15 minutes to return a list of all keys. (Note that I don't need a list of all keys while I am processing, I am just as happy to pick up data as I parse through the file.)</p>