title: Hadoop: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
body: <p>My MapReduce jobs runs ok when assembled in Eclipse with all possible Hadoop and Hive jars included in Eclipse project as dependencies. (These are the jars that come with single node, local Hadoop installation). </p>
  
  <p>Yet when trying to run the same program assembled using Maven project (see below) I get: </p>
  
  <pre><code> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
  </code></pre>
  
  <p>This exception happens when program is assembled using the following Maven project: </p>
  
  <pre><code>&lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&gt;
    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
  
    &lt;groupId&gt;com.bigdata.hadoop&lt;/groupId&gt;
    &lt;artifactId&gt;FieldCounts&lt;/artifactId&gt;
    &lt;version&gt;0.0.1-SNAPSHOT&lt;/version&gt;
    &lt;packaging&gt;jar&lt;/packaging&gt;
  
    &lt;name&gt;FieldCounts&lt;/name&gt;
    &lt;url&gt;http://maven.apache.org&lt;/url&gt;
  
    &lt;properties&gt;
      &lt;project.build.sourceEncoding&gt;UTF-8&lt;/project.build.sourceEncoding&gt;
    &lt;/properties&gt;
  
    &lt;dependencies&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;junit&lt;/groupId&gt;
        &lt;artifactId&gt;junit&lt;/artifactId&gt;
        &lt;version&gt;3.8.1&lt;/version&gt;
        &lt;scope&gt;test&lt;/scope&gt;
      &lt;/dependency&gt;
       &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
          &lt;artifactId&gt;hadoop-hdfs&lt;/artifactId&gt;
          &lt;version&gt;2.2.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
          &lt;artifactId&gt;hadoop-common&lt;/artifactId&gt;
          &lt;version&gt;2.2.0&lt;/version&gt;
      &lt;/dependency&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
      &lt;artifactId&gt;hadoop-mapreduce-client-jobclient&lt;/artifactId&gt;
      &lt;version&gt;2.2.0&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.hive.hcatalog&lt;/groupId&gt;
      &lt;artifactId&gt;hcatalog-core&lt;/artifactId&gt;
      &lt;version&gt;0.12.0&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;com.google.guava&lt;/groupId&gt;
      &lt;artifactId&gt;guava&lt;/artifactId&gt;
      &lt;version&gt;16.0.1&lt;/version&gt;
  &lt;/dependency&gt;
    &lt;/dependencies&gt;     
      &lt;build&gt;
      &lt;plugins&gt;
        &lt;plugin&gt;
          &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
          &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
          &lt;version&gt;2.3.2&lt;/version&gt;
          &lt;configuration&gt;
              &lt;source&gt;${jdk.version}&lt;/source&gt;
              &lt;target&gt;${jdk.version}&lt;/target&gt;
          &lt;/configuration&gt;
        &lt;/plugin&gt;       
    &lt;plugin&gt;
    &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
    &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;
      &lt;executions&gt;
        &lt;execution&gt;
           &lt;goals&gt;
             &lt;goal&gt;attached&lt;/goal&gt;
           &lt;/goals&gt;
           &lt;phase&gt;package&lt;/phase&gt;
           &lt;configuration&gt;
             &lt;descriptorRefs&gt;
               &lt;descriptorRef&gt;jar-with-dependencies&lt;/descriptorRef&gt;
            &lt;/descriptorRefs&gt;
            &lt;archive&gt;
              &lt;manifest&gt;
                &lt;mainClass&gt;com.bigdata.hadoop.FieldCounts&lt;/mainClass&gt;
              &lt;/manifest&gt;
            &lt;/archive&gt;
          &lt;/configuration&gt;
       &lt;/execution&gt;
    &lt;/executions&gt;
  &lt;/plugin&gt;
   &lt;/plugins&gt;
   &lt;/build&gt;       
  &lt;/project&gt;
  </code></pre>
  
  <p><strong>* Please advise where and how to find compatible Hadoop jars? *</strong></p>
  
  <p><strong>[update_1]</strong>
  I am running Hadoop 2.2.0.2.0.6.0-101</p>
  
  <p>As I have found here: <a href="https://github.com/kevinweil/elephant-bird/issues/247">https://github.com/kevinweil/elephant-bird/issues/247</a></p>
  
  <p><em>Hadoop 1.0.3: JobContext is a class</em></p>
  
  <p><em>Hadoop 2.0.0: JobContext is an interface</em></p>
  
  <p>In my pom.xml I have three jars with version 2.2.0   </p>
  
  <pre><code>hadoop-hdfs 2.2.0
  hadoop-common 2.2.0
  hadoop-mapreduce-client-jobclient 2.2.0
  hcatalog-core 0.12.0
  </code></pre>
  
  <p>The only exception is <code>hcatalog-core</code> which version is 0.12.0, I could not find <em>any more recent version of this jar</em> and I need it!</p>
  
  <p>How can I find which of these 4 jars produces <code>java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected</code> ?</p>
  
  <p>Please, give me an idea how to solve this. (The only solution I see is to compile everything from source!)</p>
  
  <p><strong>[/update_1]</strong></p>
  
  <p>Full text of my MarReduce Job:</p>
  
  <pre><code>package com.bigdata.hadoop;
  
  import java.io.IOException;
  import java.util.*;
  
  import org.apache.hadoop.conf.*;
  import org.apache.hadoop.io.*;
  import org.apache.hadoop.mapreduce.*;
  import org.apache.hadoop.util.*;
  import org.apache.hcatalog.mapreduce.*;
  import org.apache.hcatalog.data.*;
  import org.apache.hcatalog.data.schema.*;
  import org.apache.log4j.Logger;
  
  public class FieldCounts extends Configured implements Tool {
  
      public static class Map extends Mapper&lt;WritableComparable, HCatRecord, TableFieldValueKey, IntWritable&gt; {
  
          static Logger logger = Logger.getLogger("com.foo.Bar");
  
          static boolean firstMapRun = true;
          static List&lt;String&gt; fieldNameList = new LinkedList&lt;String&gt;();
          /**
           * Return a list of field names not containing `id` field name
           * @param schema
           * @return
           */
          static List&lt;String&gt; getFieldNames(HCatSchema schema) {
              // Filter out `id` name just once
              if (firstMapRun) {
                  firstMapRun = false;
                  List&lt;String&gt; fieldNames = schema.getFieldNames();
                  for (String fieldName : fieldNames) {
                      if (!fieldName.equals("id")) {
                          fieldNameList.add(fieldName);
                      }
                  }
              } // if (firstMapRun)
              return fieldNameList;
          }
  
          @Override
        protected void map( WritableComparable key,
                            HCatRecord hcatRecord,
                            //org.apache.hadoop.mapreduce.Mapper
                            //&lt;WritableComparable, HCatRecord, Text, IntWritable&gt;.Context context)
                            Context context)
              throws IOException, InterruptedException {
  
              HCatSchema schema = HCatBaseInputFormat.getTableSchema(context.getConfiguration());
  
             //String schemaTypeStr = schema.getSchemaAsTypeString();
             //logger.info("******** schemaTypeStr ********** : "+schemaTypeStr);
  
             //List&lt;String&gt; fieldNames = schema.getFieldNames();
              List&lt;String&gt; fieldNames = getFieldNames(schema);
              for (String fieldName : fieldNames) {
                  Object value = hcatRecord.get(fieldName, schema);
                  String fieldValue = null;
                  if (null == value) {
                      fieldValue = "&lt;NULL&gt;";
                  } else {
                      fieldValue = value.toString();
                  }
                  //String fieldNameValue = fieldName+"."+fieldValue;
                  //context.write(new Text(fieldNameValue), new IntWritable(1));
                  TableFieldValueKey fieldKey = new TableFieldValueKey();
                  fieldKey.fieldName = fieldName;
                  fieldKey.fieldValue = fieldValue;
                  context.write(fieldKey, new IntWritable(1));
              }
  
          }       
      }
  
      public static class Reduce extends Reducer&lt;TableFieldValueKey, IntWritable,
                                         WritableComparable, HCatRecord&gt; {
  
          protected void reduce( TableFieldValueKey key,
                                 java.lang.Iterable&lt;IntWritable&gt; values,
                                 Context context)
                                 //org.apache.hadoop.mapreduce.Reducer&lt;Text, IntWritable,
                                 //WritableComparable, HCatRecord&gt;.Context context)
              throws IOException, InterruptedException {
              Iterator&lt;IntWritable&gt; iter = values.iterator();
              int sum = 0;
              // Sum up occurrences of the given key 
              while (iter.hasNext()) {
                  IntWritable iw = iter.next();
                  sum = sum + iw.get();
              }
  
              HCatRecord record = new DefaultHCatRecord(3);
              record.set(0, key.fieldName);
              record.set(1, key.fieldValue);
              record.set(2, sum);
  
              context.write(null, record);
          }
      }
  
      public int run(String[] args) throws Exception {
          Configuration conf = getConf();
          args = new GenericOptionsParser(conf, args).getRemainingArgs();
  
          // To fix Hadoop "META-INFO" (http://stackoverflow.com/questions/17265002/hadoop-no-filesystem-for-scheme-file)
          conf.set("fs.hdfs.impl",
                  org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
          conf.set("fs.file.impl",
                  org.apache.hadoop.fs.LocalFileSystem.class.getName());
  
          // Get the input and output table names as arguments
          String inputTableName = args[0];
          String outputTableName = args[1];
          // Assume the default database
          String dbName = null;
  
          Job job = new Job(conf, "FieldCounts");
  
          HCatInputFormat.setInput(job,
                  InputJobInfo.create(dbName, inputTableName, null));
          job.setJarByClass(FieldCounts.class);
          job.setMapperClass(Map.class);
          job.setReducerClass(Reduce.class);
  
          // An HCatalog record as input
          job.setInputFormatClass(HCatInputFormat.class);
  
          // Mapper emits TableFieldValueKey as key and an integer as value
          job.setMapOutputKeyClass(TableFieldValueKey.class);
          job.setMapOutputValueClass(IntWritable.class);
  
          // Ignore the key for the reducer output; emitting an HCatalog record as
          // value
          job.setOutputKeyClass(WritableComparable.class);
          job.setOutputValueClass(DefaultHCatRecord.class);
          job.setOutputFormatClass(HCatOutputFormat.class);
  
          HCatOutputFormat.setOutput(job,
                  OutputJobInfo.create(dbName, outputTableName, null));
          HCatSchema s = HCatOutputFormat.getTableSchema(job);
          System.err.println("INFO: output schema explicitly set for writing:"
                  + s);
          HCatOutputFormat.setSchema(job, s);
          return (job.waitForCompletion(true) ? 0 : 1);
      }
  
      public static void main(String[] args) throws Exception {
          String classpath = System.getProperty("java.class.path");
          //System.out.println("*** CLASSPATH: "+classpath);       
          int exitCode = ToolRunner.run(new FieldCounts(), args);
          System.exit(exitCode);
      }
  }
  </code></pre>
  
  <p>And class for complex key:</p>
  
  <pre><code>package com.bigdata.hadoop;
  
  import java.io.DataInput;
  import java.io.DataOutput;
  import java.io.IOException;
  
  import org.apache.hadoop.io.WritableComparable;
  
  import com.google.common.collect.ComparisonChain;
  
  public class TableFieldValueKey  implements WritableComparable&lt;TableFieldValueKey&gt; {
  
        public String fieldName;
        public String fieldValue;
  
        public TableFieldValueKey() {} //must have a default constructor
        //
  
        public void readFields(DataInput in) throws IOException {
          fieldName = in.readUTF();
          fieldValue = in.readUTF();
        }
  
        public void write(DataOutput out) throws IOException {
          out.writeUTF(fieldName);
          out.writeUTF(fieldValue);
        }
  
        public int compareTo(TableFieldValueKey o) {
          return ComparisonChain.start().compare(fieldName, o.fieldName)
              .compare(fieldValue, o.fieldValue).result();
        }
  
      }
  </code></pre>