title: GC overhead while running pig job, after hadoop job ends
body: <p>I'm running a very simple pig script (pig 0.14, Hadoop 2.4) : </p>
  
  <pre><code>customers = load '/some/hdfs/path' using SomeUDFLoader();
  customers2 = foreach (group customers by customer_id) generate FLATTEN(group) as customer_id, MIN(dw_customer.date) as date;
  store customers2 into '/hdfs/output' using PigStorage(',');
  </code></pre>
  
  <p>This launches a map-reduce job of ~60000 mappers, and 999 reducers.</p>
  
  <p><strong>After the map-reduce job has finished it's work</strong> ( I know becuase the output has been written, and the job manager says the job has succeeded ), There is a long pause and I get the following error in the pig output :</p>
  
  <pre><code>2015-11-24 11:45:29,394 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at *********
  2015-11-24 11:45:29,403 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
  2015-11-24 11:46:03,533 [Service Thread] INFO  org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call- Usage threshold init = 698875904(682496K) used = 520031456(507843K) committed = 698875904(682496K) max = 698875904(682496K)
  2015-11-24 11:46:04,473 [Service Thread] INFO  org.apache.pig.impl.util.SpillableMemoryManager - first memory handler call - Collection threshold init = 698875904(682496K) used = 575405920(561919K) committed = 698875904(682496K) max = 698875904(682496K)
  2015-11-24 11:47:36,255 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. GC overhead limit exceeded
  </code></pre>
  
  <p>The stack trace looks something like (each time the exception in is another function ):</p>
  
  <pre><code>Pig Stack Trace
  ---------------
  ERROR 2998: Unhandled internal error. Java heap space
  
  java.lang.OutOfMemoryError: Java heap space
      at org.apache.hadoop.mapreduce.v2.api.records.impl.pb.CounterGroupPBImpl.initCounters(CounterGroupPBImpl.java:136)
      at org.apache.hadoop.mapreduce.v2.api.records.impl.pb.CounterGroupPBImpl.getAllCounters(CounterGroupPBImpl.java:121)
      at org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
      at org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:367)
      at org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:388)
      at org.apache.hadoop.mapred.ClientServiceDelegate.getTaskReports(ClientServiceDelegate.java:448)
      at org.apache.hadoop.mapred.YARNRunner.getTaskReports(YARNRunner.java:551)
      at org.apache.hadoop.mapreduce.Job$3.run(Job.java:533)
      at org.apache.hadoop.mapreduce.Job$3.run(Job.java:531)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
      at org.apache.hadoop.mapreduce.Job.getTaskReports(Job.java:531)
      at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getTaskReports(HadoopShims.java:235)
      at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:352)
      at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:233)
      at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:165)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:360)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280)
      at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
      ...
  </code></pre>
  
  <p>My set of SET statements in the pig script :</p>
  
  <pre><code>SET mapreduce.map.java.opts '-server -Xmx6144m -Djava.net.preferIPv4Stack=true -Duser.timezone=UTC'
  SET mapreduce.reduce.java.opts '-server -Xmx6144m -Djava.net.preferIPv4Stack=true -Duser.timezone=UTC'
  SET mapreduce.map.memory.mb '8192'
  SET mapreduce.reduce.memory.mb '8192'
  SET mapreduce.map.speculative 'true'
  SET mapreduce.reduce.speculative 'true'
  SET mapreduce.jobtracker.maxtasks.perjob '100000'
  SET mapreduce.job.split.metainfo.maxsize '-1'
  </code></pre>
  
  <p>Why is this happening, and how can I fix that ?</p>
  
  <p>Thanks in advance for any help.</p>