title: very fast text file processing (C++)
body: <p>i wrote an application which processes data on the GPU. Code works well, but i have the problem that the reading part of the input file (~3GB, text) is the bottleneck of my application. (The read from the HDD is fast, but the processing line by line is slow).</p>
  
  <p>I read a line with getline() and copy line 1 to a vector, line2 to a vector and skip lines 3 and 4. And so on for the rest of the 11 mio lines.</p>
  
  <p>I tried several approaches to get the file at the best time possible:</p>
  
  <p>Fastest method I found is using boost::iostreams::stream</p>
  
  <p>Others were:</p>
  
  <ul>
  <li>Read the file as gzip, to minimize IO, but is slower than directly
  reading it. </li>
  <li>copy file to ram by read(filepointer, chararray, length)
  and process it with a loop to distinguish the lines (also slower than boost)</li>
  </ul>
  
  <p>Any suggestions how to make it run faster?</p>
  
  <pre><code>void readfastq(char *filename, int SRlength, uint32_t blocksize){
      _filelength = 0; //total datasets (each 4 lines)
      _SRlength = SRlength; //length of the 2. line
      _blocksize = blocksize;
  
      boost::iostreams::stream&lt;boost::iostreams::file_source&gt;ins(filename);
      in = ins;
  
      readNextBlock();
  }
  
  
  void readNextBlock() {
      timeval start, end;
      gettimeofday(&amp;start, 0);
  
      string name;
      string seqtemp;
      string garbage;
      string phredtemp;
  
      _seqs.empty();
      _phred.empty();
      _names.empty();
      _filelength = 0;
  
              //read only a part of the file i.e the first 4mio lines
      while (std::getline(in, name) &amp;&amp; _filelength&lt;_blocksize) {
          std::getline(in, seqtemp);
          std::getline(in, garbage);
          std::getline(in, phredtemp);
  
          if (seqtemp.size() != _SRlength) {
              if (seqtemp.size() != 0)
                  printf("Error on read in fastq: size is invalid\n");
          } else {
              _names.push_back(name);
  
              for (int k = 0; k &lt; _SRlength; k++) {
  
                  //handle special letters
                                      if(seqtemp[k]== 'A') ...
                                      else{
                  _seqs.push_back(5);
                                      }
  
              }
              _filelength++;
          }
      }
  </code></pre>
  
  <p><strong>EDIT:</strong></p>
  
  <p>The source-file is downloadable under <a href="https://docs.google.com/open?id=0B5bvyb427McSMjM2YWQwM2YtZGU2Mi00OGVmLThkODAtYzJhODIzYjNhYTY2" rel="noreferrer">https://docs.google.com/open?id=0B5bvyb427McSMjM2YWQwM2YtZGU2Mi00OGVmLThkODAtYzJhODIzYjNhYTY2</a></p>
  
  <p>I changed the function readfastq to read the file, because of some pointer problems. So if you call <code>readfastq</code> the <code>blocksize</code> (in lines) must be bigger than the number of lines to read.</p>
  
  <p><strong>SOLUTION:</strong></p>
  
  <p>I found a solution, which get the time for read in the file from 60sec to 16sec. I removed the inner-loop which handeles the special characters and do this in GPU. This decreases the read-in time and only minimal increases the GPU running time.</p>
  
  <p>Thanks for your suggestions.</p>
  
  <pre><code>void readfastq(char *filename, int SRlength) {
      _filelength = 0;
      _SRlength = SRlength;
  
      size_t bytes_read, bytes_expected;
  
      FILE *fp;
      fp = fopen(filename, "r");
  
      fseek(fp, 0L, SEEK_END); //go to the end of file
      bytes_expected = ftell(fp); //get filesize
      fseek(fp, 0L, SEEK_SET); //go to the begining of the file
  
      fclose(fp);
  
      if ((_seqarray = (char *) malloc(bytes_expected/2)) == NULL) //allocate space for file
          err(EX_OSERR, "data malloc");
  
  
      string name;
      string seqtemp;
      string garbage;
      string phredtemp;
  
      boost::iostreams::stream&lt;boost::iostreams::file_source&gt;file(filename);
  
  
      while (std::getline(file, name)) {
          std::getline(file, seqtemp);
          std::getline(file, garbage);
          std::getline(file, phredtemp);
  
          if (seqtemp.size() != SRlength) {
              if (seqtemp.size() != 0)
                  printf("Error on read in fastq: size is invalid\n");
          } else {
              _names.push_back(name);
  
              strncpy( &amp;(_seqarray[SRlength*_filelength]), seqtemp.c_str(), seqtemp.length()); //do not handle special letters here, do on GPU
  
              _filelength++;
          }
      }
  }
  </code></pre>