title: Debug failed shuffles in hadoop map reduces
body: <p>I am seeing as the size of the input file increase failed shuffles increases and job complete time increases non linearly.</p>
  
  <p>eg.</p>
  
  <pre><code>75GB took 1h
  86GB took 5h
  </code></pre>
  
  <p>I also see average shuffle time increase 10 fold</p>
  
  <p>eg.</p>
  
  <pre><code>75GB 4min
  85GB 41min
  </code></pre>
  
  <p>Can someone point me a direction to debug this?</p>