title: Compression performance related to chunk size in hdf5 files
body: <p>I would like to ask a question about the performance of compression
  which is related to chunk size of hdf5 files.</p>
  
  <p>I have 2 hdf5 files on hand, which have the following properties.
  They both only contain one dataset, called "data".</p>
  
  <p>File A's "data":</p>
  
  <ol>
  <li>Type: HDF5 Scalar Dataset</li>
  <li>No. of Dimensions: 2</li>
  <li>Dimension Size: <strong>5094125 x 6</strong></li>
  <li>Max. dimension size: Unlimited x Unlimited</li>
  <li>Data type: 64-bit floating point</li>
  <li>Chunking: <strong>10000 x 6</strong></li>
  <li>Compression: GZIP level = 7</li>
  </ol>
  
  <p>File B's "data":</p>
  
  <ol>
  <li>Type: HDF5 Scalar Dataset</li>
  <li>No. of Dimensions: 2</li>
  <li>Dimension Size: <strong>6720 x 1000</strong></li>
  <li>Max. dimension size: Unlimited x Unlimited</li>
  <li>Data type: 64-bit floating point</li>
  <li>Chunking: <strong>6000 x 1</strong></li>
  <li>Compression: GZIP level = 7</li>
  </ol>
  
  <p><strong>File A's size: 
  HDF5----19 MB
  CSV-----165 MB</strong></p>
  
  <p><strong>File B's size: 
  HDF5----60 MB
  CSV-----165 MB</strong></p>
  
  <p>Both of them shows great compression on data stored when comparing to csv files.
  However, the compression rate of file A is about 10% of original csv,
  while that of file B is only about 30% of original csv.</p>
  
  <p>I have tried different chunk size to make file B as small as possible, but it seems that 30% is the optimum compression rate. I would like to ask why file A can achieve a greater compression while file B cannot. </p>
  
  <p>If file B can also achieve, what should the chunk size be?</p>
  
  <p>Is that any rule to determine the optimum chunk size of HDF5 for compression purpose?</p>
  
  <p>Thanks!</p>