title: How can I generate a smaller HDF5 file?
body: <p>I am working on an image classification problem with a dataset of about 1,000,000 images summing up to a total size of ~36 GB. I am trying to train a CNN model with this data on a machine with 256GB RAM (and 4 x 16GB VRAM). </p>
  
  <p>To solve the memory issues that I was facing with directly feeding the images to the program, I tried converting the raw image data to HDF5 format. However, the size of the HDF5 file kept growing and the converter crashed when it ran out of disk space. The HDF5 file sitting on the disk is at 2.2TB!!</p>
  
  <p>Is this size expected? How can I convert this to a (preferably much) smaller file?</p>