title: GlusterFS read directory performance in replicated setup - what's wrong? how to optimize?
body: <p>I'm currently playing around with a two-node cluster with Gluster 3.5 in replication mode. This is to get some insight in the system before trying to implement a real 3-node cluster on server hardware.</p>
  
  <p>The test hardware is nothing high-end: a Intel Atom D2550 and a Intel i5 connected together via a Ethernet cross-cable on the Gbit ports.</p>
  
  <p>In the Gluster file system there are about 20,000 mostly small files (basically a Debian installation), which is similar to the real world usage it will need to handle later (on different hardware).</p>
  
  <p>Since some old software will run on the brick, that unfortunately requires to poll periodically over most of these files, latency when polling file stats is a factor.</p>
  
  <p>I did a simple test (GlusterFS mounted on the Gluster node itself):</p>
  
  <pre><code># time find | wc -l
  22174
  
  real    0m18.542s
  user    0m0.224s
  sys     0m0.789s
  </code></pre>
  
  <p>From what i know, this maybe is so slow because GlusterFS has to poll the other node on each <code>stat</code>.</p>
  
  <p><em>When polling directly on the brick storage directory, starting with the second trial I get timings in the range of 0.16 seconds (as expected, everything is probably read from cache).</em></p>
  
  <p>However, when I shut down the other node, so that there is <strong>only one node left</strong>, I get pretty similar results:</p>
  
  <pre><code># time find | wc -l
  22174
  
  real    0m16.445s
  user    0m0.213s
  sys     0m0.702s
  </code></pre>
  
  <p>How can that be? What's slowing down Gluster in this case?</p>
  
  <p>In general, how can I minimize read latency in GlusterFS redundant setup? It would not be a problem if polling the file system dirlist would temporarily lag behind the real situation during recovery after a crash, if that could improve dirlist performance..</p>