title: Pig Latin: Load multiple files from a date range (part of the directory structure)
body: <p>I have the following scenario-</p>
  
  <p>Pig version used 0.70</p>
  
  <p>Sample HDFS directory structure:</p>
  
  <pre><code>/user/training/test/20100810/&lt;data files&gt;
  /user/training/test/20100811/&lt;data files&gt;
  /user/training/test/20100812/&lt;data files&gt;
  /user/training/test/20100813/&lt;data files&gt;
  /user/training/test/20100814/&lt;data files&gt;
  </code></pre>
  
  <p>As you can see in the paths listed above, one of the directory names is a date stamp.</p>
  
  <p><strong>Problem:</strong> I want to load files from a date range say from 20100810 to 20100813.</p>
  
  <p>I can pass the 'from' and 'to' of the date range as parameters to the Pig script but how do I make use of these parameters in the LOAD statement. I am able to do the following</p>
  
  <pre><code>temp = LOAD '/user/training/test/{20100810,20100811,20100812}' USING SomeLoader() AS (...);
  </code></pre>
  
  <p>The following works with hadoop:</p>
  
  <pre><code>hadoop fs -ls /user/training/test/{20100810..20100813}
  </code></pre>
  
  <p>But it fails when I try the same with LOAD inside the pig script. How do I make use of the parameters passed to the Pig script to load data from a date range? </p>
  
  <p>Error log follows:</p>
  
  <pre><code>Backend error message during job submission
  -------------------------------------------
  org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input splits for: hdfs://&lt;ServerName&gt;.com/user/training/test/{20100810..20100813}
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:269)
          at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:858)
          at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:875)
          at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
          at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:793)
          at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:752)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062)
          at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:752)
          at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:726)
          at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
          at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
          at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
          at java.lang.Thread.run(Thread.java:619)
  Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input Pattern hdfs://&lt;ServerName&gt;.com/user/training/test/{20100810..20100813} matches 0 files
          at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
          at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:258)
          ... 14 more
  
  
  
  Pig Stack Trace
  ---------------
  ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input splits for: hdfs://&lt;ServerName&gt;.com/user/training/test/{20100810..20100813}
  
  org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias test
          at org.apache.pig.PigServer.openIterator(PigServer.java:521)
          at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:544)
          at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
          at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:162)
          at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138)
          at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
          at org.apache.pig.Main.main(Main.java:357)
  Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input splits for: hdfs://&lt;ServerName&gt;.com/user/training/test/{20100810..20100813}
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:169)
  </code></pre>
  
  <p>Do I need to make use of a higher language like Python to capture all date stamps in the range and pass them to LOAD as a comma separated list?</p>
  
  <p>cheers</p>