title: Hive Job failed with return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask and Query Performance
body: <p>Every day I have a hive job that compute some aggregations for each quarter of hour for two months of data. It's resulting in submitting something like 5760 jobs to Tez.</p>
  
  <p>The job failed with the following error in the stderr :</p>
  
  <pre><code>FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
  </code></pre>
  
  <p>The error could occurred after 2300 - 2500 tez jobs. Just before this error, There is a lot of the following logs in the Yarn Logs :</p>
  
  <p><code>2015-12-10 21:53:35,286 INFO [TezChild] task.ContainerReporter: Sleeping for 200ms before retrying getTask again. Got null now. Next getTask sleep message after 2000ms</code></p>
  
  <p>And the execution time of a single job is dramatically expending, from 20s to 100s.</p>
  
  <p>I don't have any clue about my issue and I can't find anything else in yarn, Hadoop, hive, or tez logs (no exceptions, nothing marked as an error).</p>
  
  <p>So I have two questions :
  Where can I find more information in logs or something else that could help me to resolve this issue ?</p>
  
  <p>Currently we use :</p>
  
  <ul>
  <li>The latest version of Azure HDInsight 3.2</li>
  <li>Jobs are submitted to the cluster with the C# SDK</li>
  <li>Hive jobs use tez</li>
  </ul>
  
  <p><strong>Question 2 :</strong>
  I'm pretty sure that we do not do our aggregations in a good way. For each aggregation (i.e for each quarter), we should retrieve the precedent value of a row. I hoped to use the LAG function, but we could not pass a predicate for finding the precedent value (we need the precedent value that is not greater than the current value). So we could not find an other way than generate a query for each quarter that we need to compute.
  Is someone know how we can do that in a single hive query ?</p>
  
  <p>Thanks in advance for any help,
  Best regards</p>