title: Integration testing Hive jobs
body: <p>I'm trying to write a non-trivial Hive job using the Hive Thrift and JDBC interfaces, and I'm having trouble setting up a decent JUnit test. By non-trivial, I mean that the job results in at least one MapReduce stage, as opposed to only dealing with the metastore.</p>
  
  <p>The test should fire up a Hive server, load some data into a table, run some non-trivial query on that table, and check the results.</p>
  
  <p>I've wired up a Spring context according to the <a href="http://static.springsource.org/spring-hadoop/docs/1.0.0.RELEASE/reference/html/hive.html" rel="noreferrer" title="Spring Hive context">Spring reference</a>. However, the job fails on the MapReduce phase, complaining that no Hadoop binary exists:</p>
  
  <blockquote>
    <p>java.io.IOException: Cannot run program "/usr/bin/hadoop" (in
    directory "/Users/yoni/opower/workspace/intellij_project_root"):
    error=2, No such file or directory</p>
  </blockquote>
  
  <p>The problem is that the Hive Server is running in-memory, but relies upon local installation of Hive in order to run. For my project to be self-contained, I need the Hive services to be embedded, including the HDFS and MapReduce clusters. I've tried starting up a Hive server using the same Spring method and pointing it at <a href="http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.203.0/org/apache/hadoop/hdfs/MiniDFSCluster.html" rel="noreferrer">MiniDFSCluster</a> and <a href="http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapred/MiniMRCluster.html" rel="noreferrer">MiniMRCluster</a>, similar to the pattern used in the Hive <a href="http://hive.apache.org/docs/r0.9.0/api/org/apache/hadoop/hive/ql/QTestUtil.html" rel="noreferrer">QTestUtil</a> source and in <a href="http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/testapidocs/org/apache/hadoop/hbase/HBaseTestingUtility.html" rel="noreferrer">HBaseTestUtility</a>. However, I've not been able to get that to work.</p>
  
  <p>After three days of trying to wrangle Hive integration testing, I thought I'd ask the community: </p>
  
  <ol>
  <li>How do you recommend I integration test Hive jobs?</li>
  <li>Do you have a working JUnit example for integration testing Hive jobs using in-memory HDFS, MR, and Hive instances?</li>
  </ol>
  
  <p>Additional resources I've looked at:</p>
  
  <ul>
  <li><a href="http://dev.bizo.com/2011/04/hive-unit-testing.html" rel="noreferrer" title="Unit testing Hive">Hive Unit Testing tutorial</a></li>
  <li><a href="https://github.com/SpringSource/spring-hadoop-samples/tree/master/samples/hive" rel="noreferrer" title="Spring Hive example">the Spring Hive example</a></li>
  </ul>
  
  <p>Edit:
  I am fully aware that working against a Hadoop cluster - whether local or remote - makes it possible to run integration tests against a full-stack Hive instance. The problem, as stated, is that this is not a viable solution for effectively testing Hive workflows.</p>