title: Hadoop setInputPathFilter error
body: <p>I am using Hadoop 0.20.2 (that cannot be changed) and I want to add a filter to my input path. The data looks as follows:</p>
  
  <pre><code>/path1/test_a1
  /path1/test_a2
  /path1/train_a1
  /path1/train_a2
  </code></pre>
  
  <p>and I only want to process all files with <em>train</em> in them.</p>
  
  <p>A look at the FileInputFormat class suggests to use:</p>
  
  <pre><code> FileInputFormat.setInputPathFilter(Job job, Class&lt;? extends PathFilter&gt; filter)
  </code></pre>
  
  <p>and this is where my problem starts, since <a href="http://hadoop.apache.org/docs/r0.20.2/api/org/apache/hadoop/fs/PathFilter.html" rel="nofollow">PathFilter</a> is an interface - of course, I can extend the interface but then I still do not have an implementation. So instead, I implemented the interface:</p>
  
  <pre><code>class TrainFilter implements PathFilter
  {
     boolean accept(Path path)
     {
        return path.toString().contains("train");
     }
  }
  </code></pre>
  
  <p>When I use TrainFilter as PathFilter the code compiles, however when I run it, I get an exception as the input path is screwed up. Without setting the filter, my code runs through all files that are below /path1, however, when setting the filter, it throws the error:</p>
  
  <pre><code>InvalidInputException: Input path does not exist hdfs://localhost:9000/path1
  </code></pre>
  
  <p>Here is how I set it up in the driver code:</p>
  
  <pre><code>job.setMapperClass(....class);
  job.setInputFormatClass(....class);
  job.setMapOutputKeyClass(...class);
  job.setMapOutputValueClass(Text.class);
  job.setOutputKeyClass(IntWritable.class);
  job.setOutputValueClass(IntWritable.class);
  
  FileInputFormat.setInputPathFilter(job, TrainFilter.class);
  FileInputFormat.addInputPath(job, new Path("/path1/"));
  FileOutputFormat.setOutputPath(job, new Path("/path2/"));
  job.waitForCompletion(true);
  </code></pre>
  
  <p>Any suggestions of what I am doing wrong here?</p>
  
  <p>EDIT: I found the problem. The first call to the PathFilter is always the directory itself (/path1) and since it does not contain ("train"), the directory itself is invalid and thus the exception is thrown. Which brings me to another question: how can I test if an arbitrary path is a directory? For all I know, I need a reference to the FileSystem, which is not one of the default parameters of PathFilter. </p>