title: How to Test HDFS I/O Throughput
body: <p>I have a Java program that reads files and write the contents to new files using the HDFS data input/output stream. My goal is find out the I/O throughput of my HDFS. Below is the code fragment that does the read/write and timing:</p>
  
  <pre><code>long start = System.currentTimeMillis();
  FSDataInputStream in = fs.open( new Path(input) );
  FSDataOutputStream out = fs.create( new Path(output), true);
  
  while ((bytesRead = in.read(buffer)) &gt; 0) {
    out.write(buffer, 0, bytesRead);
    data += bytesRead;
  }
  
  in.close();
  out.close();
  long end = System.currentTimeMillis();
  
  System.out.println("Copy data " + data + " Bytes in " +
  ((double)end-start) + " millisecond");
  </code></pre>
  
  <p>I expected the time to copy the file proportional to the file size. But when I ran the program for files from 5MB to 50MB, the result did not show this correlation:</p>
  
  <pre><code>Copy data 5242880 Bytes in 844.0 millisecond
  Copy data 10485760 Bytes in 733.0 millisecond
  Copy data 15728640 Bytes in 901.0 millisecond
  Copy data 20971520 Bytes in 1278.0 millisecond
  Copy data 26214400 Bytes in 1304.0 millisecond
  Copy data 31457280 Bytes in 1543.0 millisecond
  Copy data 36700160 Bytes in 2091.0 millisecond
  Copy data 41943040 Bytes in 1934.0 millisecond
  Copy data 47185920 Bytes in 1847.0 millisecond
  Copy data 52428800 Bytes in 3222.0 millisecond
  </code></pre>
  
  <p>My question is: Why the copy time is not proportional to the file size? Do I use the wrong methods? Any feedback will be appreciated.</p>
  
  <p>My Hadoop is run on the Pseudo-Distributed Operation mode and I clear the cache using command:</p>
  
  <pre><code>sudo sh -c "sync; echo 3 &gt; /proc/sys/vm/drop_caches"
  </code></pre>
  
  <p>every time before running the program.</p>