title: Compressing HDF5 files with H5Py
body: <p>I'm passing thousands of .csv containing <code>time</code> and <code>amplitude</code> to a .hdf5 file. To give an example I used a small amount of .csv files which correspond to a total of ~11MB.</p>
  
  <p>After passing all the .csv to hdf5, the latter has a size of ~36MB (without using <code>compression="gzip"</code>).</p>
  
  <p>By using <code>compression="gzip"</code> the file size is around 38MB.</p>
  
  <p>I understand that hdf5 is compressing the dataset only, that is, the numpy array in my case (~500 rows with float number).</p>
  
  <p>To make a comparison, I was saving all the .csv in a .json file, compressing it and then reading. I chose hdf5 due to memory issues since the json file is loaded entirely into memory with a footprint 2x to Xx times larger than the file size.</p>
  
  <p>This is how I append to a dataset in a .hdf5 file.</p>
  
  <pre class="lang-py prettyprint-override"><code>def hdf5_dump_dataset(hdf5_filename, hdf5_data, dsetname):
          with h5py.File(hdf5_filename, 'a') as f:
              dset = f.create_dataset(dsetname, data=hdf5_data, compression="gzip", chunks=True, maxshape=(None,))
  </code></pre>
  
  <p>This is how I read a dataset from a .hdf5 file.</p>
  
  <pre class="lang-py prettyprint-override"><code>def hdf5_load_dataset(hdf5_filename, dsetname):
          with h5py.File(hdf5_filename, 'r') as f:
              dset = f[dsetname]
              return dset[...]
  </code></pre>
  
  <p>The folder structre with the .csv files:</p>
  
  <pre><code>root/
      folder_1/
          file_1.csv
          file_X.csv  
      folder_X/
          file_1.csv
          file_X.csv  
  </code></pre>
  
  <p>Inside each .csv file: </p>
  
  <pre><code>time, amplitude
  1.000e-08, -1.432e-07
  1.001e-08, 7.992e-07
  1.003e-08, -1.838e-05
  1.003e-08, 2.521e-05
  </code></pre>
  
  <p>script to save the .csv contents in a hdf5 file:</p>
  
  <pre><code># csv_dict is a dict() with all folders and csv files as keys
  # ex. csv_dict['folder_1']['file_1']  (without the .csv extension)
  
  for folder in csv_dict:
      for file in csv_dict[folder]:
          path_waveform = f"{folder}/{file}.csv"
          time, amplitude = self.read_csv_return_list_of_time_amplitude(path_waveform)
  
          hdf5_dump_dataset(path_hdf5_waveforms, amplitude, '/'.join([folder, file, 'amplitude']))
  
          hdf5_dump_dataset(path_hdf5_waveforms, time, '/'.join([folder, file, 'time']))
  </code></pre>
  
  <p>For each .csv file in each folder I have a dataset for the <code>time</code> and for the <code>amplitude</code>. The structure of the hdfile is like this:</p>
  
  <pre><code>folder1/file_1/time
  folder1/file_1/amplitude
  </code></pre>
  
  <p>where</p>
  
  <pre><code>time = np.array([1.000e-08, 1.001e-08, 1.003e-08, ...])  # 500 items
  amplitude = np.array([-1.432e-07, 7.992e-07, -1.838e-05, ...])  # 500 items
  </code></pre>
  
  <p>My question is: Is there a way to compress the whole hdf5 file?</p>