title: Can Flatbuffers take advantage of 0's in vectors? Or are other wavelets better than the Haar transform?
body: <p>I'm serializing some data and want to make the file size as small as possible without losing the essential details of the data.  The first step for me was to save the data in a binary format instead of ASCII and I decided to try Flatbuffers.  Previously when the data were stored as text files, they were about 400 mb.  Using the schema shown below, the file is about 200 mb.  So that's a nice decrease in size, but smaller would of course be better.  The data consist of 1 of the ControlParams, 82 of the ControlData, and the intensity vector takes up most of the space, being a matrix with a size of about 128x5000.  We are already around the theoretical binary size of 128x5000*82 * 4 bytes per float ~ 200 mb.  The matrices are pretty dense in general, but here and there I can see rows that are zero.  Can Flatbuffers take advantage of these zeros to reduce the file size further?  Perhaps there are other inefficiencies that someone can spot in the schema, since I just am getting started with Flatbuffers?</p>
  
  <p>Another way to go about reducing the file size might be investigating different wavelets to compress the original intensities with.  I'm using the Haar transform now because I was able to make a C++ function to do this, and found that a compression of 2x or possibly 4x was possible.  I might like to investigate other wavelets, but would like to know if others have tried different wavelets compared to Haar and found they were able to use fewer coefficients with them.</p>
  
  <pre><code>namespace RTSerialization;
  
  table ControlParams{
      extractStepSizeDa:float = 1.0005;
      smooth:bool = false;
      haarLevel:int = 10;
      deltaTimeSec:float;
  }
  
  table ControlData{
      mzAxis:[float];
      timeSec:[float];
      intensities:[float];
      scanFilter:string;
  }
  
  table ControlParamsAndData{
      params:ControlParams;
      dataSet:[ControlData];
  }
  
  root_type ControlParamsAndData;
  </code></pre>