title: Kafka Quickstart: What Dependencies do I need?
body: <p>I am working through the kafka quickstart:</p>
  
  <p><a href="http://kafka.apache.org/07/quickstart.html">http://kafka.apache.org/07/quickstart.html</a></p>
  
  <p>and the basic Consumer Group example:</p>
  
  <p><a href="https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example">https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example</a></p>
  
  <p>I have coded up the Consumer and ConsumerThreadPool as above: </p>
  
  <pre><code>import kafka.consumer.KafkaStream;
  import kafka.consumer.ConsumerIterator;
  
  public class Consumer implements Runnable {
  
      private KafkaStream m_stream;
      private Integer m_threadNumber;
  
      public Consumer(KafkaStream a_stream, Integer a_threadNumber) {
          m_threadNumber = a_threadNumber;
          m_stream = a_stream;
      }
  
      public void run() {
          ConsumerIterator&lt;byte[], byte[]&gt; it = m_stream.iterator();
          while (it.hasNext()) {
              System.out.println("Thread " + m_threadNumber + ": " + new String(it.next().message()));
  
          }
          System.out.println("Shutting down Thread: " + m_threadNumber);
      }
  }
  </code></pre>
  
  <p>A couple of other facets: I am using spring to manage my zookeeper:</p>
  
  <pre><code>import javax.inject.Named;
  import java.util.Properties;
  import kafka.consumer.ConsumerConfig;
  import org.springframework.context.annotation.Bean;
  import org.springframework.context.annotation.ComponentScan;
  import org.springframework.context.annotation.Configuration;
  
  @Configuration
  @ComponentScan("com.truecar.inventory.worker.core")
  public class AppConfig {
  
      @Bean
      @Named("consumerConfig")
      private static ConsumerConfig createConsumerConfig() {
          String zookeeperAddress = "127.0.0.1:2181";
          String groupId = "inventory";
          Properties props = new Properties();
          props.put("zookeeper.connect", zookeeperAddress);
          props.put("group.id", groupId);
          props.put("zookeeper.session.timeout.ms", "400");
          props.put("zookeeper.sync.time.ms", "200");
          props.put("auto.commit.interval.ms", "1000");
          return new ConsumerConfig(props);
      }
  }
  </code></pre>
  
  <p>And I am compiling with Maven and the OneJar Maven plugin.  However, I compile and then run the resulting one jar I get the following error: </p>
  
  <pre><code>Aug 26, 2013 6:15:41 PM org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider registerDefaultFilters
  INFO: JSR-330 'javax.inject.Named' annotation found and supported for component scanning
  Exception in thread "main" java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at com.simontuffs.onejar.Boot.run(Boot.java:340)
  at com.simontuffs.onejar.Boot.main(Boot.java:166)
  Caused by: java.lang.NoClassDefFoundError: scala/ScalaObject
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:792)
  at com.simontuffs.onejar.JarClassLoader.defineClass(JarClassLoader.java:803)
  at com.simontuffs.onejar.JarClassLoader.findClass(JarClassLoader.java:710)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at com.simontuffs.onejar.JarClassLoader.loadClass(JarClassLoader.java:630)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at java.lang.Class.getDeclaredMethods0(Native Method)
  at java.lang.Class.privateGetDeclaredMethods(Class.java:2521)
  at java.lang.Class.getDeclaredMethods(Class.java:1845)
  at org.springframework.core.type.StandardAnnotationMetadata.getAnnotatedMethods(StandardAnnotationMetadata.java:180)
  at org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(ConfigurationClassParser.java:222)
  at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:165)
  at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:140)
  at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:282)
  at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:223)
  at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:630)
  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:461)
  at org.springframework.context.annotation.AnnotationConfigApplicationContext.&lt;init&gt;(AnnotationConfigApplicationContext.java:73)
  at com.truecar.inventory.worker.core.consumer.ConsumerThreadPool.&lt;clinit&gt;(ConsumerThreadPool.java:31)
  at com.truecar.inventory.worker.core.application.Starter.main(Starter.java:20)
  ... 6 more
  Caused by: java.lang.ClassNotFoundException: scala.ScalaObject
  at com.simontuffs.onejar.JarClassLoader.findClass(JarClassLoader.java:713)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at com.simontuffs.onejar.JarClassLoader.loadClass(JarClassLoader.java:630)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 27 more
  </code></pre>
  
  <p>Now, I know little about Kafka, and nothing about Scala.  How do I fix this?  What should i try next?  Is this a known issue?  Do I need other dependencies?  Here is the kafka version in my pom.xml:</p>
  
  <pre><code>&lt;dependency&gt;
      &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
      &lt;artifactId&gt;kafka_2.9.2&lt;/artifactId&gt;
      &lt;version&gt;0.8.0-beta1&lt;/version&gt;
  &lt;/dependency&gt; 
  </code></pre>
  
  <p>Update:  I contacted the Kafka dev mailing list, and they let me know some specific version requirements for the scala dependencies.  However, there is also an undocumented log4j dependency, which results in another runtime, not compile time, exception.  </p>
  
  <pre><code>Exception in thread "main" java.lang.reflect.InvocationTargetException
  Caused by: java.lang.NoSuchMethodError: ch.qos.logback.classic.Logger.filterAndLog(Ljava/lang/String;Lorg/slf4j/Marker;Lch/qos/logback/classic/Level;Ljava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
  at org.apache.log4j.Category.log(Category.java:333)
  at org.apache.commons.logging.impl.Log4JLogger.debug(Log4JLogger.java:177)
  </code></pre>
  
  <p>Another Update:</p>
  
  <p>I found the correct log4j dependency:</p>
  
  <pre><code>    &lt;dependency&gt;
          &lt;groupId&gt;log4j&lt;/groupId&gt;
          &lt;artifactId&gt;log4j&lt;/artifactId&gt;
          &lt;version&gt;1.2.17&lt;/version&gt;
      &lt;/dependency&gt;
  </code></pre>
  
  <p>But now I am met with an even more cryptic runtime exception:</p>
  
  <pre><code>Exception in thread "main" java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at com.simontuffs.onejar.Boot.run(Boot.java:340)
  at com.simontuffs.onejar.Boot.main(Boot.java:166)
  Caused by: java.lang.NoClassDefFoundError: org/I0Itec/zkclient/IZkStateListener
  at kafka.javaapi.consumer.ZookeeperConsumerConnector.&lt;init&gt;(ZookeeperConsumerConnector.scala:64)
  at kafka.javaapi.consumer.ZookeeperConsumerConnector.&lt;init&gt;(ZookeeperConsumerConnector.scala:66)
  at kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:100)
  at kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala)
  </code></pre>
  
  <p>At this point I got the WTF kind of feeling.  So I added another dependency:</p>
  
  <pre><code>    &lt;dependency&gt;
          &lt;groupId&gt;com.101tec&lt;/groupId&gt;
          &lt;artifactId&gt;zkclient&lt;/artifactId&gt;
          &lt;version&gt;0.3&lt;/version&gt;
      &lt;/dependency&gt;
  </code></pre>
  
  <p>But this exposed yet another runtime exception:</p>
  
  <pre><code>Exception in thread "main" java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at com.simontuffs.onejar.Boot.run(Boot.java:340)
  at com.simontuffs.onejar.Boot.main(Boot.java:166)
  Caused by: java.lang.NoClassDefFoundError: com/yammer/metrics/core/Gauge
  at kafka.consumer.ZookeeperConsumerConnector.createFetcher(ZookeeperConsumerConnector.scala:146)
  at kafka.consumer.ZookeeperConsumerConnector.&lt;init&gt;(ZookeeperConsumerConnector.scala:113)
  at kafka.javaapi.consumer.ZookeeperConsumerConnector.&lt;init&gt;(ZookeeperConsumerConnector.scala:64)
  at kafka.javaapi.consumer.ZookeeperConsumerConnector.&lt;init&gt;(ZookeeperConsumerConnector.scala:66)
  at kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:100)
  at kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala)
  </code></pre>
  
  <p>I am hoping to be able to get this baby example up and running, but maybe this is the price to pay for using beta products?  Maybe I should switch to Apache Active MQ.  But that sounds less fun.  Am I missing something?</p>