title: FileChannel close is too slow. Will pre allocating space improve things?
body: <p>I use a FileChannel to write 4GB files to a spin disc and although I have tweaked the buffer size to maximise write speed and flush the channel every second the file channel close can take 200 ms.  This is enough time that the queue that I read from overflows and starts dropping packets.</p>
  
  <p>I use a direct byte buffer, but I am struggling to understand what is happening here.  I have removable discs and write caching has been disabled so I would not expect the OS to be buffering the data?</p>
  
  <p>The benchmark speed of the discs is around 80 MB/Sec, but I am seeing the long file channel close times even when writing at speeds of ~ 40 MB/Sec.</p>
  
  <p>I appreciate that as the discs fills then write performance will decrease, but these discs are empty.</p>
  
  <p>Is there any tweaks I can do to remove the long delay when closing the file channel.  Should I be allocating the file space upfront and I write the file with a .lock extension and then do a rename once the file has been completed? </p>
  
  <p>Just hoping someone who has done high throughput IO can provide some pointers as to possible options above and beyond what is usually documented when writing files using NIO.</p>
  
  <p>The code is below and I cannot see anything immediately wrong.</p>
  
  <p>public final class DataWriter implements Closeable {</p>
  
  <pre><code>private static final Logger LOG = Logger.getLogger("DataWriter");
  
  private static final long MB = 1024 * 1024;
  
  private final int flushPeriod;
  
  private FileOutputStream fos;
  private FileChannel fileChannel;
  private long totalBytesWritten;
  private long lastFlushTime;
  private final ByteBuffer buffer;
  private final int bufferSize;
  private final long startTime;
  private long totalPackets = 0;
  private final String fileName;
  
  public DataWriter(File recordFile, int bSize, int flushPeriod) throws IOException {
      this.flushPeriod = flushPeriod;
      if (!recordFile.createNewFile()) {
          throw new IllegalStateException("Record file has not been created");
      }
  
      totalBytesWritten = 0;
      fos = new FileOutputStream(recordFile);
      fileChannel = fos.getChannel();
      buffer = ByteBuffer.allocateDirect(bSize);
      bufferSize = bSize;
      startTime = System.currentTimeMillis();
      this.fileName = recordFile.getAbsolutePath();
  }
  
  /**
   * Appends the supplied ByteBuffer to the main buffer if there is space 
   * @param packet
   * @return
   * @throws IOException
   */
  public int write(ByteBuffer packet) throws IOException {
      int bytesWritten = 0;
      totalPackets++;
  
      //If the buffer cannot accommodate the supplied buffer then write straight out
      if(packet.limit() &gt; buffer.capacity()) {
          bytesWritten = writeBuffer(packet);
          totalBytesWritten += bytesWritten;  
      } else {
  
         //write the currently filled buffer if no space exists to accomodate the current buffer
          if(packet.limit() &gt; buffer.remaining()) {
             buffer.flip();
             bytesWritten = writeBuffer(buffer);
             totalBytesWritten += bytesWritten;   
          } 
  
          buffer.put(packet);                 
      }
  
      if(System.currentTimeMillis()-lastFlushTime &gt; flushPeriod) {
          fileChannel.force(true);
          lastFlushTime=System.currentTimeMillis();
      }
  
      return bytesWritten;
  
  }
  
  public long getTotalBytesWritten() {
      return totalBytesWritten;
  }
  
  /**
   * Writes the buffer and then clears it
   * @throws IOException
   */
  private int writeBuffer(ByteBuffer byteBuffer) throws IOException {
      int bytesWritten = 0;
      while(byteBuffer.hasRemaining()) {
          bytesWritten += fileChannel.write(byteBuffer);
      }
  
      //Reset the buffer ready for writing
      byteBuffer.clear();
      return bytesWritten;
  }
  
  @Override
  public void close() throws IOException {
      //Write the buffer if data is present
      if(buffer.position() != 0) {
          buffer.flip();
          totalBytesWritten += writeBuffer(buffer);
          fileChannel.force(true);
      }
  
      long time = System.currentTimeMillis() - startTime;
      if(LOG.isDebugEnabled()) {
        LOG.debug(  totalBytesWritten + " bytes written in " + (time / 1000d) + " seconds using ByteBuffer size ["+bufferSize/1024+"] KB");
        LOG.debug(  (totalBytesWritten / MB) / (time / 1000d) + " MB per second written to file " + fileName);
        LOG.debug(  "Total packets written ["+totalPackets+"] average packet size ["+totalBytesWritten / totalPackets+"] bytes");
      }
  
      if (fos != null) {
          fos.close();
          fos = null;
      }
  }
  </code></pre>
  
  <p>}</p>