title: Java FileChannel Vs BufferedReader - Spring Batch - Reader
body: <p>We process huge files (sometimes 50 GB each file). The application reads this one file and based on the business logic, it will write multiple output files (4-6).</p>
  
  <p>The records in the file are of variable length and each field in a record is a delimiter separated.</p>
  
  <p>Going by the understanding that reading a file using FileChannel with a ByteBuffer was always better than using a BufferedReader.readLine and then using a split by the delimiter.</p>
  
  <ul>
  <li>BufferSizes tried 10240(10KB) and even more</li>
  <li>Commit interval - 5000, 10000 etc</li>
  </ul>
  
  <p>Below is how we used file channel to read:</p>
  
  <ul>
  <li>Read byte by byte. Check if the read byte is a new line char(10) -
  which means end of line. </li>
  <li>check for delimiter bytes. capture the bytes read in a byte array(we initialized this byte array with a maximum field size of 350 bytes) until delimiter bytes are encountered.</li>
  <li>convert these bytes read until this time, to String using UTF-8 encoding - new String(byteArr, 0, index,"UTF-8") to be specific - index is the number of bytes read until delimiter.</li>
  </ul>
  
  <p>Using this method of reading the file using FileChannel took 57 minutes to process the file.</p>
  
  <p>We want to decrease this time and tried using BufferredReader.readLine() and then use a split by delimiter,  to see how it fares.</p>
  
  <p>And shockingly the same file completed processing only in 7 minutes.</p>
  
  <p>What's the catch here? Why FileChannel is taking more time than a buffered reader and then using a string split.</p>
  
  <p>I was always under the assumption that ReadLine and Split combination will have a big performance impact?</p>
  
  <p>Can any one throw light on if I was using FileChannel in a wrong way? One </p>
  
  <p>Thanks in advance. Hope I have summarized the issue properly. </p>
  
  <p>The below is sample code : </p>
  
  <pre><code>while (inputByteBuffer.hasRemaining() &amp;&amp; (b = inputByteBuffer.get()) != 0){
          boolean endOfField = false;
          if (b == 10){
              break;
          }
          else{
              if (b == 94){//^
                  if (!inputByteBuffer.hasRemaining()){
                      inputByteBuffer.clear();
                      noOfBytes = inputFileChannel.read(inputByteBuffer);
                      inputByteBuffer.flip();
                  }
                  if (inputByteBuffer.hasRemaining()){
                      byte b2 = inputByteBuffer.get();
                      if (b2 == 124){//|
                          if (!inputByteBuffer.hasRemaining()){
                              inputByteBuffer.clear();
                              noOfBytes = inputFileChannel.read(inputByteBuffer);
                              inputByteBuffer.flip();
                          }
  
                          if (inputByteBuffer.hasRemaining()){
                              byte b3 = inputByteBuffer.get();
                              if (b3 == 94){//^
                                  String field = new String(fieldBytes, 0, index, encoding);
                                  if(fieldIndex == -1){
                                      fields = new String[sizeFromAConfiguration];
                                  }else{
                                      fields[fieldIndex] = field;
                                  }
  
                                  fieldBytes = new byte[maxFieldSize];
                                  endOfField = true;
                                  fieldIndex++;
                              }
                              else{
                                  fieldBytes = addFieldBytes(fieldBytes, b, index);
                                  index++;
                                  fieldBytes = addFieldBytes(fieldBytes, b2, index);
                                  index++;
                                  fieldBytes = addFieldBytes(fieldBytes, b3, index);
                              }
                          }
                          else{
                              endOfFile = true;
                              //fields.add(new String(fieldBytes, 0, index, encoding));
                              fields[fieldIndex] = new String(fieldBytes, 0, index, encoding);
                              fieldBytes = new byte[maxFieldSize];
                              endOfField = true;
                          }
                      }else{
                          fieldBytes = addFieldBytes(fieldBytes, b, index);
                          index++;
                          fieldBytes = addFieldBytes(fieldBytes, b2, index);
  
                      }
                  }else{
                      endOfFile = true;
                      fieldBytes = addFieldBytes(fieldBytes, b, index);
                  }
              }
              else{
                  fieldBytes = addFieldBytes(fieldBytes, b, index);
              }
          }
  
          if (!inputByteBuffer.hasRemaining()){
              inputByteBuffer.clear();
              noOfBytes = inputFileChannel.read(inputByteBuffer);
              inputByteBuffer.flip();
          }
  
          if (endOfField){
              index = 0;
          }
          else{
              index++;
          }
  
      }
  </code></pre>