title: Importing multi-level directories of logs in hadoop/pig
body: <p>We store our logs in S3, and one of our (Pig) queries would grab three different log types. Each log type is in sets of subdirectories based upon type/date. For instance:</p>
  
  <pre><code>/logs/&lt;type&gt;/&lt;year&gt;/&lt;month&gt;/&lt;day&gt;/&lt;hour&gt;/lots_of_logs_for_this_hour_and_type.log*
  </code></pre>
  
  <p>my query would want to load all three types of logs, for a give time. For instance:</p>
  
  <pre><code>type1 = load 's3:/logs/type1/2011/03/08' as ...
  type2 = load 's3:/logs/type2/2011/03/08' as ...
  type3 = load 's3:/logs/type3/2011/03/08' as ...
  result = join type1 ..., type2, etc...
  </code></pre>
  
  <p>my queries would then run against all of these logs.</p>
  
  <p>What is the most efficient way to handle this?</p>
  
  <ol>
  <li>Do we need use the bash script expansion? Not sure if this works with multi directories, and I doubt it would be efficient (or even possible) if there were 10k logs to load.</li>
  <li>Do we create a service to aggregate all of the logs and push them to hdfs directly?</li>
  <li>Custom java/python importers? </li>
  <li>Other thoughts?</li>
  </ol>
  
  <p>If you could leave some example code, if appropriate, as well, that would be helpful.</p>
  
  <p>Thanks</p>