title: How do I store data in multiple, partitioned files on HDFS using Pig
body: <p>I've got a pig job that analyzes a large number of log files and generates a relationship between a group of attributes and a bag of IDs that have those attributes.  I'd like to store that relationship on HDFS, but I'd like to do so in a way that is friendly for other Hive/Pig/MapReduce jobs to operate on the data, or subsets of the data without having to ingest the full output of my pig job, as that is a significant amount of data.</p>
  
  <p>For example, if the schema of my relationship is something like:</p>
  
  <pre><code>relation: {group: (attr1: long,attr2: chararray,attr3: chararray),ids: {(id: chararray)}}
  </code></pre>
  
  <p>I'd really like to be able to partition this data, storing it in a file structure that looks like:</p>
  
  <pre><code>/results/attr1/attr2/attr3/file(s)
  </code></pre>
  
  <p>where the attrX values in the path are the values from the group, and the file(s) contain only ids.  This would allow me to easily subset my data for subsequent analysis without duplicating data.  </p>
  
  <p>Is such a thing possible, even with a custom StoreFunc?  Is there a different approach that I should be taking to accomplish this goal?</p>
  
  <p>I'm pretty new to Pig, so any help or general suggestions about my approach would be greatly appreciated.</p>
  
  <p>Thanks in advance.</p>