title: HDFS performance for small files
body: <p>I'm new to Hadoop. Recently I'm trying to process (only read) many <strong>small</strong> files on hdfs/hadoop. The average file size is about <strong>1 kb</strong> and the number of files is more than 10M. The program must be written in C++ due to some limitations.</p>
  
  <p>This is just a performance evaluation so I only use 5 machines for data nodes. Each of the data node have 5 data disks.</p>
  
  <p>I wrote a small <strong>C++ project to read the files directly from hard disk</strong>(not from HDFS) to build the performance base line. The program will create 4 reading threads for each disk. The performance result is to have about 14MB/s per disk. Total throughput is about 14MB/s * 5 * 5 = <strong>350MB</strong>/s (14MB/s * 5 disks * 5 machines ).</p>
  
  <p>However, when this program ( still using C++, dynamically linked to libhdfs.so, creating 4*5*5=100 threads) <strong>reads files from hdfs</strong> cluster, the throughput is about only <strong>55MB/s</strong>.</p>
  
  <p>If this programming is triggered in mapreduce (hadoop streamming, 5 jobs, each have 20 threads, total number of threads is still 100), the throughput goes down to about 45MB/s. (I guess it's slow down by some bookkeeping process).</p>
  
  <p>I'm wondering what is the reasonable performance HDFS can prvoide. As you can see, comparing with native code, the data throughput is only about <strong>1/7</strong>. Is it the problem of my config? Or HDFS limitation? Or Java limitation? What's the best way for my scenario? Will sequence file help (much)? What is the reasonable throughput comparing to native IO read we can expect?</p>
  
  <p>Here's some of my config:</p>
  
  <p>NameNode heap size 32G.</p>
  
  <p>Job/Task node heap size 8G.</p>
  
  <p>NameNode Handler Count: 128</p>
  
  <p>DataNode Handler Count: 8</p>
  
  <p>DataNode Maximum Number of Transfer Threads: 4096</p>
  
  <p>1GBps ethernet.</p>
  
  <p>Thanks.</p>