title: Hadoop map throw "Filesystem closed" exception
body: <p>I'm running a MapReduce task against Wikipedia dump with history <strong>using XmlInputFormat for parsing the XML.</strong>
  "xxx_m_000053_0" always stop at 70% before it's kill due to time out.</p>
  
  <p><strong>in the console:</strong></p>
  
  <blockquote>
    <p>xxx_m_000053_0 failed to report status for 300 seconds. Killing!</p>
  </blockquote>
  
  <p>I increase the timeout to 2 hours. It didn't work.</p>
  
  <p><br />
  <strong>In  xxx_m_000053_0 log file:</strong></p>
  
  <blockquote>
    <p>Processing split: hdfs://localhost:8020/user/martin/history/history.xml:3556769792+67108864</p>
  </blockquote>
  
  <p>I was expecting something wrong in history.xml in offset [3556769792,3623878656]. <a href="https://stackoverflow.com/questions/18841786/specify-hadoop-process-split">I split the file from this offset and run it in hadoop</a>. It worked... (???)</p>
  
  <p><strong>In  xxx_m_000053_0 log file:</strong></p>
  
  <pre><code>java.io.IOException: Filesystem closed
      at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
      at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
      at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
      at java.io.FilterInputStream.close(FilterInputStream.java:155)
      **at com.doduck.wikilink.history.XmlInputFormat$XmlRecordReader.close(XmlInputFormat.java:109)**
      at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
      at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)
  2013-09-17 13:13:32,248 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
  2013-09-17 13:13:32,248 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@54e9a7c2
  org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/file.out
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127)
      at org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:69)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1645)
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1328)
      at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:698)
      at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1793)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
      at org.apache.hadoop.mapred.Child.main(Child.java:249)
  </code></pre>
  
  <p><br />
  So I'm thinking it might be a configuration problem? Why is my file system stop?
  Something wrong with <a href="https://github.com/apache/mahout/blob/ad84344e4055b1e6adff5779339a33fa29e1265d/examples/src/main/java/org/apache/mahout/classifier/bayes/XmlInputFormat.java" rel="nofollow noreferrer">XmlInputFormat</a> ?</p>
  
  <p><br /></p>
  
  <hr>
  
  <p><br /></p>
  
  <p>My <strong>empty</strong> mapper:</p>
  
  <pre><code>@Override
  public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException
  {
   //nothing to do...
  }
  </code></pre>
  
  <p><br />
  My Main:</p>
  
  <pre><code>    public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException {
      Configuration conf = new Configuration();
      conf.set("xmlinput.start", "&lt;page&gt;");
      conf.set("xmlinput.end", "&lt;/page&gt;");
  
      Job job = new Job(conf, "wikipedia link history");
      job.setJarByClass(Main.class);
      job.setMapperClass(Map.class);
      job.setReducerClass(Reduce.class);
      job.setInputFormatClass(XmlInputFormat.class);
  
      job.setMapOutputKeyClass(Text.class);
      job.setMapOutputValueClass(IntWritable.class);
      job.setOutputKeyClass(Text.class);
      job.setOutputValueClass(Text.class);
      job.setOutputFormatClass(TextOutputFormat.class);
  
      FileInputFormat.addInputPath(job, new Path(args[0]));
      FileOutputFormat.setOutputPath(job, new Path(args[1]));
      boolean result = job.waitForCompletion(true);
      System.exit(result ? 0 : 1);
  }
  </code></pre>
  
  <p><br />
  hdfs-site.xml:</p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;dfs.replication&lt;/name&gt;
    &lt;value&gt;1&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p><br />
  mapred-site.xml:</p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;mapred.job.tracker&lt;/name&gt;
    &lt;value&gt;localhost:9001&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
      &lt;name&gt;mapred.tasktracker.map.tasks.maximum&lt;/name&gt;
      &lt;value&gt;2&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
      &lt;name&gt;mapred.tasktracker.reduce.tasks.maximum&lt;/name&gt;
      &lt;value&gt;2&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
      &lt;name&gt;mapred.child.java.opts&lt;/name&gt;
      &lt;value&gt;-Xmx9216m&lt;/value&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
      &lt;name&gt;mapred.task.timeout&lt;/name&gt;
      &lt;value&gt;300000&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p><br />
  My core-site.xml</p>
  
  <pre><code>&lt;configuration&gt;
  &lt;property&gt;
      &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;
      &lt;value&gt;/Volumes/WD/hadoop/tmp&lt;/value&gt;
      &lt;description&gt;A base for other temporary directories.&lt;/description&gt;
  &lt;/property&gt;
  &lt;property&gt;
      &lt;name&gt;fs.default.name&lt;/name&gt;
      &lt;value&gt;hdfs://localhost:8020&lt;/value&gt;
  &lt;/property&gt;
  </code></pre>
  
  <p></p>