title: Pig keeps trying to connect to job history server (and fails)
body: <p>I'm running a Pig job that fails to connect to the Hadoop job history server.</p>
  
  <p>The task (usually any task with <code>GROUP BY</code>) runs for a while and then it starts with a message like:</p>
  
  <pre><code>2015-04-21 19:05:22,825 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
  2015-04-21 19:05:26,721 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2015-04-21 19:05:29,721 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  </code></pre>
  
  <p>It then continues for a while retrying the connection. Sometimes it precedes further with the job. Othertimes it throws this exception:</p>
  
  <pre><code>2015-04-21 19:05:55,822 [main] WARN  org.apache.pig.tools.pigstats.mapreduce.MRJobStats - Unable to get job counters
  java.io.IOException: java.io.IOException: java.net.NoRouteToHostException: No Route to Host from  cluster-01/10.10.10.11 to 0.0.0.0:10020 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see:  http://wiki.apache.org/hadoop/NoRouteToHost
      at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getCounters(HadoopShims.java:132)
      at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addCounters(MRJobStats.java:284)
      at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:235)
      at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:165)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:360)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280)
  </code></pre>
  
  <p>I found this question <a href="https://stackoverflow.com/questions/17930644/connection-error-in-apache-pig">here</a> but in my case the job history server is started. If I run netstat, I find:</p>
  
  <pre><code>tcp        0      0 0.0.0.0:10020           0.0.0.0:*               LISTEN      12073/java       off (0.00/0/0)
  </code></pre>
  
  <p>Where 12073 is ...</p>
  
  <pre><code>12073 pts/4    Sl     0:07 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Dproc_historyserver -Xmx1000m -Djava.library.path=/data/hadoop/hadoop/lib -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/data/hadoop/hadoop-2.3.0/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/data/hadoop/hadoop-2.3.0 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/data/hadoop/hadoop/logs -Dhadoop.log.file=mapred-hadoop-historyserver-cluster-01.log -Dhadoop.root.logger=INFO,RFA -Dmapred.jobsummary.logger=INFO,JSA -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer
  </code></pre>
  
  <p>I tried opening the port 10200 in case it was a firewall issue:</p>
  
  <pre><code>ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:10020
  </code></pre>
  
  <p>... but no luck.</p>
  
  <p>After a few minutes, some of the tasks just arbitrarily continue to the next part.</p>
  
  <p>I'm using Hadoop 2.3 and Pig 0.14.</p>
  
  <p>My question is:</p>
  
  <p>1) What are the possible reasons why Pig cannot connect to the job history server (JHS) given that the JHS is running on the same port that Pig looks for it?</p>
  
  <p>... or failing that ...</p>
  
  <p>2) Is there any way to just tell Pig to stop trying to connect to the JHS and continue with the task?</p>