title: Yarn local-dirs - per node setup
body: <p>I've had a series of devops issues from time to time on our production cluster. Every now and then, <code>/</code> partition gets overwhelmed on couple of nodes. Long story short, it turns out that these nodes had 1 instead of 2 data drives. This would not be an issue if we don't have a following setup on our cluster:</p>
  
  <pre><code>   &lt;property&gt;       
        &lt;name&gt;yarn.nodemanager.local-dirs&lt;/name&gt;       
        &lt;value&gt;/data1/hadoop/yarn/local,/data2/hadoop/yarn/local&lt;/value&gt; 
     &lt;/property&gt;
  </code></pre>
  
  <p>Some devops or whoever, noticing there are no /data2 partitions on the <em>smaller</em> nodes, came up with the idea to simply go with <code>/</code> partition. Since <code>/</code> is 16GB, some of the more data-demanding jobs quickly fill the thing.</p>
  
  <p>Now, my question: does yarn support per-node setup of yarn.nodemanager.local-dirs? </p>
  
  <p>I resolved the problem by removing <code>/data2/hadoop/yarn/local</code> from the story, but it doesn't feel perfect.</p>
  
  <p>We're using HDP 2.6.4.</p>
  
  <p>Thx!</p>