title: Optimizing disk IO
body: <p>I have a piece of code that analyzes streams of data from very large (10-100GB) binary files. It works well, so it's time to start optimizing, and currently disk IO is the biggest bottleneck.</p>
  
  <p>There are two types of files in use. The first type of file consists of a stream of 16-bit integers, which must be scaled after I/O to convert to a floating point value which is physically meaningful. I read the file in chunks, and I read in the chunks of data by reading one 16-bit code at a time, performing the required scaling, and then storing the result in an array. Code is below:</p>
  
  <pre><code>int64_t read_current_chimera(FILE *input, double *current,
                               int64_t position, int64_t length, chimera *daqsetup)
  {
      int64_t test;
      uint16_t iv;
  
      int64_t i;
      int64_t read = 0;
  
      if (fseeko64(input, (off64_t)position * sizeof(uint16_t), SEEK_SET))
      {
          return 0;
      }
  
      for (i = 0; i &lt; length; i++)
      {
          test = fread(&amp;iv, sizeof(uint16_t), 1, input);
          if (test == 1)
          {
              read++;
              current[i] = chimera_gain(iv, daqsetup);
          }
          else
          {
              perror("End of file reached");
              break;
          }
      }
      return read;
  }
  </code></pre>
  
  <p>The chimera_gain function just takes a 16-bit integer, scales it and returns the double for storage. </p>
  
  <p>The second file type contains 64-bit doubles, but it contains two columns, of which I only need the first. To do this I fread pairs of doubles and discard the second one. The double must also be endian-swapped before use. The code I use to do this is below:</p>
  
  <pre><code>int64_t read_current_double(FILE *input, double *current, int64_t position, int64_t length)
  {
      int64_t test;
      double iv[2];
  
      int64_t i;
      int64_t read = 0;
  
      if (fseeko64(input, (off64_t)position * 2 * sizeof(double), SEEK_SET))
      {
          return 0;
      }
  
      for (i = 0; i &lt; length; i++)
      {
          test = fread(iv, sizeof(double), 2, input);
          if (test == 2)
          {
              read++;
              swapByteOrder((int64_t *)&amp;iv[0]);
              current[i] = iv[0];
          }
          else
          {
              perror("End of file reached: ");
              break;
          }
      }
      return read;
  }
  </code></pre>
  
  <p>Can anyone suggest a method of reading these file types that would be significantly faster than what I am currently doing?</p>