title: Specific memory limitations of Pig LOAD statement?
body: <p><strong>Simple question:</strong> </p>
  
  <p>What is the memory limitation of the Pig LOAD statement? </p>
  
  <p><strong>More detailed question:</strong> </p>
  
  <p>Is there any way to relate available physical resources (disk, RAM, CPU) to the maximum size of a directory that a Pig LOAD statement can handle? </p>
  
  <p><strong>Scenario:</strong></p>
  
  <p>A research project is using a Pig script that is trying to load a directory containing 12,000+ files with a total size of 891GB in a single Pig LOAD statement, copied below. The files are gzipped WAT files which describe, in raw text, a collection of web pages. When run, the job appears to crash/hang/freeze our cluster every time. Since we are all new to Hadoop, the suspicion has been on resources and configuration until I finally was able to review the code. </p>
  
  <pre><code>-- load data from I_WATS_DIR 
  Orig = LOAD '$I_WATS_DIR' USING org.archive.hadoop.ArchiveJSONViewLoader('Envelope.ARC-Header-    Metadata.Target-URI','var2...','var3...','var4...{1,2,3,4} as
  (src:chararray,timestamp:chararray,html_base:chararray,relative:chararray,path:chararray,text:chararray,alt:chararray); 
  </code></pre>
  
  <p><strong>Details:</strong> </p>
  
  <p>CLUSTER </p>
  
  <p>1 front end node, 16 cores, 64GB RAM, 128GB swap, NameNode </p>
  
  <p>3 compute nodes, 16 cores, 128GB RAM, 128GB swap, DataNode </p>
  
  <p>TEST JOB 1 </p>
  
  <p>Same script referenced above, loading a directory with 1 file</p>
  
  <p>Resident memory reported 1.2GB </p>
  
  <p>Input: 138MB </p>
  
  <p>Output: 207MB </p>
  
  <p>Reduce input records: 1,630,477 </p>
  
  <p>Duration: 4m 11s </p>
  
  <p>TEST JOB 2 </p>
  
  <p>Same script, 17 files </p>
  
  <p>Resident memory: 16.4GB </p>
  
  <p>Input: 3.5GB </p>
  
  <p>Output: 1.3GB </p>
  
  <p>Reduce input records: 10,648,807 </p>
  
  <p>Duration: 6m 48s </p>
  
  <p>TEST JOB 3 </p>
  
  <p>Same script, 51 files </p>
  
  <p>Resident memory: 41.4GB </p>
  
  <p>Input: 10.9GB </p>
  
  <p>Output: not recorded </p>
  
  <p>Reduce input records: 31,968,331 </p>
  
  <p>Duration: 6m 18s </p>
  
  <p><strong>Final thoughts:</strong></p>
  
  <p>This is a 4 node cluster with nothing else running on it, fully dedicated to Cloudera Hadoop CDH4, running this 1 job only. Hoping this is all the info people need to answer my original question! I strongly suspect that some sort of file parsing loop that loads 1 file at a time is the solution, but I know even less about Pig than I do about Hadoop. I do have a programming/development background, but in this case I am the sys admin, not the researcher or programmer. </p>