title: Hadoop: job runs okay on smaller set of data but fails with large dataset
body: <p>I have a following situation  </p>
  
  <p>I have 3 machines cluster with following confirguration.</p>
  
  <p><code>Master</code>  </p>
  
  <pre><code>Usage of /:   91.4% of 74.41GB 
  MemTotal:       16557308 kB
  MemFree:          723736 kB 
  </code></pre>
  
  <p><code>Slave 01</code></p>
  
  <pre><code>Usage of /:   52.9% of 29.76GB
  MemTotal:       16466220 kB 
  MemFree:         5320860 kB
  </code></pre>
  
  <p><code>Slave 02</code></p>
  
  <pre><code>Usage of /:   19.0% of 19.84GB
  MemTotal:       16466220 kB
  MemFree:         6173564 kB
  </code></pre>
  
  <p><strong>hadoop/conf/core-site.xml</strong></p>
  
  <pre><code>&lt;?xml version="1.0"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  
  &lt;!-- Put site-specific property overrides in this file. --&gt;
  
  &lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;
    &lt;value&gt;/work/app/hadoop/tmp&lt;/value&gt;
    &lt;description&gt;A base for other temporary directories.&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;fs.default.name&lt;/name&gt;
    &lt;value&gt;hdfs://master:54310&lt;/value&gt;
    &lt;description&gt;The name of the default file system.  A URI whose
    scheme and authority determine the FileSystem implementation.  The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class.  The uri's authority is used to
    determine the host, port, etc. for a filesystem.&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.datanode.max.xcievers&lt;/name&gt;
    &lt;value&gt;4096&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p><strong>hadoop/conf/mapred-site.xml</strong></p>
  
  <pre><code>&lt;?xml version="1.0"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  
  &lt;!-- Put site-specific property overrides in this file. --&gt;
  
  &lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;mapred.job.tracker&lt;/name&gt;
    &lt;value&gt;master:54311&lt;/value&gt;
    &lt;description&gt;The host and port that the MapReduce job tracker runs
    at.  If "local", then jobs are run in-process as a single map
    and reduce task.
    &lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;mapred.reduce.tasks&lt;/name&gt;
    &lt;value&gt;1&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;mapred.map.tasks&lt;/name&gt;
    &lt;value&gt;100&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;mapred.task.timeout&lt;/name&gt;
    &lt;value&gt;0&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;mapred.child.java.opts&lt;/name&gt;
    &lt;value&gt;-Xmx512m&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p><strong>hadoop/conf/hdfs-site.xml</strong></p>
  
  <pre><code>&lt;?xml version="1.0"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  
  &lt;!-- Put site-specific property overrides in this file. --&gt;
  
  &lt;configuration&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.replication&lt;/name&gt;
    &lt;value&gt;3&lt;/value&gt;
    &lt;description&gt;Default block replication.
    The actual number of replications can be specified when the file is created.
    The default is used if replication is not specified in create time.
    &lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;dfs.datanode.socket.write.timeout&lt;/name&gt;
    &lt;value&gt;0&lt;/value&gt;
  &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <ul>
  <li>I have over 2 million XML documents(each document size ~ 400 KB)</li>
  <li><code>map</code> tasks opens each of these xmls and emit them as <code>JSON</code></li>
  <li><code>reduce</code> task gets each of these <code>JSON</code> as string, applies transformation and emits it  </li>
  <li>no. of <code>map</code> tasks - 100</li>
  <li>no. of <code>reduce</code> tasks - 01</li>
  <li>The entire job runs good when <code>number of documents = 10,000</code></li>
  <li>when <code>number of documents = 278262</code>, the job fails and I see various issues as following  </li>
  </ul>
  
  <p><strong>On WebUI</strong></p>
  
  <p><em>on slave-01, slave-02</em>  </p>
  
  <pre><code>java.lang.Throwable: Child Error
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  Caused by: java.io.IOException: Task process exit with nonzero status of 255.
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
  </code></pre>
  
  <p><em>On master</em></p>
  
  <pre><code>java.lang.RuntimeException: java.io.IOException: Spill failed
      at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)
      at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:545)
      at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:132)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
      at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:261)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
      at org.apache.hadoop.mapred.Child.main(Child.java:255)
  Caused by: java.io.IOException: Spill failed
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
      at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:592)
      at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:381)
  Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/spill1.out
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127)
      at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:121)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1392)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
  </code></pre>
  
  <hr>
  
  <pre><code>java.lang.Throwable: Child Error
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  Caused by: java.io.IOException: Creation of /work/app/hadoop/tmp/mapred/local/userlogs/job_201207220051_0001/attempt_201207220051_0001_m_000004_2 failed.
      at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:102)
      at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
      at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
  
  -------
  java.lang.Throwable: Child Error
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
  Caused by: java.io.IOException: Creation of /work/app/hadoop/tmp/mapred/local/userlogs/job_201207220051_0001/attempt_201207220051_0001_m_000004_2.cleanup failed.
      at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:102)
      at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71)
      at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316)
      at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228)
  </code></pre>
  
  <p>When I go and check logs in <code>slaves</code>, this is what I found in <code>hadoop-hduser-datanode-hadoop-01.log</code></p>
  
  <pre><code>2012-07-22 09:26:52,795 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-5384386931827098009_1010 src: /10.0.0.81:51402 dest: /10.0.0.82:50010
  2012-07-22 09:26:52,800 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver constructor. Cause is 
  2012-07-22 09:26:52,800 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-5384386931827098009_1010 received exception java.io.IOException: Unexpected problem in creating temporary file for blk_-5384386931827098009_1010.  File /work/app/hadoop/tmp/dfs/data/tmp/blk_-5384386931827098009 should not be present, but is.
  2012-07-22 09:26:52,800 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.0.0.82:50010, storageID=DS-735951984-127.0.1.1-50010-1342943517618, infoPort=50075, ipcPort=50020):DataXceiver
  java.io.IOException: Unexpected problem in creating temporary file for blk_-5384386931827098009_1010.  File /work/app/hadoop/tmp/dfs/data/tmp/blk_-5384386931827098009 should not be present, but is.
          at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:426)
          at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:404)
          at org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1249)
          at org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1138)
          at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.&lt;init&gt;(BlockReceiver.java:99)
          at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:299)
          at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:107)
          at java.lang.Thread.run(Thread.java:662)
  </code></pre>
  
  <p><strong>Please help me understand what is that I need to do inorder to resolve this issue?</strong></p>