title: Hadoop cluster with some nodes of low disk space~
body: <p>i'm now holding a cluster with 12 nodes. Some of them, specifically 8 nodes, have enough disk space. But the other 4 only have little space to use.</p>
  
  <p>However, the other 4 nodes have still high RAM and CPU configuration. So my intention is to utilize the resources. But now, when i ran a algorithm, SlopeOne, the map will output so many intermediate data and store them on the disk. Thus there are some errors, which I pasted under this description.</p>
  
  <p>I wonder:</p>
  
  <ol>
  <li>If a node find it can't store the data locally, will it try to store the data to other nodes with enough disk space?</li>
  <li>If a single node failed to store the data locally, will it start the work once again?</li>
  <li>If some nodes with enough disk space finish a map job first, will it continue to run the jobs which were distributed to the low disk space ones?</li>
  <li>I know I can set a parameter, which can limit the local space usage, and if a node exceed that limitation, the jobtracker won't give more jobs to that node. But will this method just keeps the node staying there without working?</li>
  <li>Any suggestions that I can utilize the resources as well as keep the errors away?</li>
  </ol>
  
  <p>Appreciate any ideas.</p>
  
  <pre><code>java.io.IOException: Spill failed
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:860)
      at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
      at slopeone.SlopeOneTrainer$SlopeOneTrainMapper.map(SlopeOneTrainer.java:71)
      at slopeone.SlopeOneTrainer$SlopeOneTrainMapper.map(SlopeOneTrainer.java:1)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
      at org.apache.hadoop.mapred.Child.main(Child.java:170)
  Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for taskTracker/jobcache/job_201104070658_0006/attempt_201104070658_0006_m_000000_0/output/spill897.out
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
      at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1221)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:686)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1173)
  </code></pre>