title: How to resolve a zookeeper authentication failure when using Kafka with Kerberos
body: <p>I am trying to use SASL between my kafka broker and zookeeper. When I start the kafka server </p>
  
  <pre><code>KAFKA_OPTS="-Djava.security.auth.login.config=/home/kafka/kafka/config/kafka_server_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf" \
      ./kafka-server-start.sh ../config/server.properties
  </code></pre>
  
  <p>I get the following error:</p>
  
  <pre><code>INFO TGT refresh thread started. (org.apache.zookeeper.Login)
  DEBUG Client principal is "kafkabroker1/kafka.eigenroute.com@EIGENROUTE.COM". (org.apache.zookeeper.Login)
  DEBUG Server principal is "krbtgt/EIGENROUTE.COM@EIGENROUTE.COM". (org.apache.zookeeper.Login)
  INFO TGT valid starting at:        Sat Dec 16 00:32:52 EST 2017 (org.apache.zookeeper.Login)
  INFO TGT expires:                  Sat Dec 16 10:32:52 EST 2017 (org.apache.zookeeper.Login)
  INFO TGT refresh sleeping until: Sat Dec 16 08:55:41 EST 2017 (org.apache.zookeeper.Login)
  INFO Opening socket connection to server devel-2.sjml.com/173.243.38.81:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
  DEBUG Closing ZkClient... (org.I0Itec.zkclient.ZkClient)
  INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
  DEBUG Closing ZooKeeper connected to zookeeper.eigenroute.com:2181 (org.I0Itec.zkclient.ZkConnection)
  DEBUG Closing session: 0x0 (org.apache.zookeeper.ZooKeeper)
  DEBUG Closing client for session: 0x0 (org.apache.zookeeper.ClientCnxn)
  WARN Client session timed out, have not heard from server in 6004ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
  DEBUG An exception was thrown while closing send thread for session 0x0 : Client session timed out, have not heard from server in 6004ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
  DEBUG Ignoring exception during shutdown input (org.apache.zookeeper.ClientCnxnSocketNIO)
  java.net.SocketException: Socket is not connected
  </code></pre>
  
  <p>My questions are: what is going on? And how to do I fix this?</p>
  
  <p>Below are my configuration files. The first is <code>server.properties</code>:</p>
  
  <pre><code># server.properties
  broker.id=0
  delete.topic.enable=true
  
  listeners=SASL_PLAINTEXT://kafka.eigenroute.com:9092
  sasl.mechanism.inter.broker.protocol=GSSAPI
  sasl.enabled.mechanisms=GSSAPI
  sasl.kerberos.service.name=kafkabroker1
  inter.broker.listener.name=SASL_PLAINTEXT
  
  authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
  super.users=User:kafkabroker1
  
  ...
  
  zookeeper.connect=zookeeper.eigenroute.com:2181
  zookeeper.connection.timeout.ms=6000
  </code></pre>
  
  <p>Here is <code>kafka_server_jaas.conf</code></p>
  
  <pre><code>KafkaServer {
          com.sun.security.auth.module.Krb5LoginModule required
          refreshKrb5Config=true
          useKeyTab=true
          storeKey=true
          keyTab="/home/kafka/keytabs/kafka_broker1.keytab"
          principal="kafkabroker1/kafka.eigenroute.com@EIGENROUTE.COM";
  };
  
  Client {
          com.sun.security.auth.module.Krb5LoginModule required
          useTicketCache=true
          refreshKrb5Config=true
          useKeyTab=true
          storeKey=true
          keyTab="/home/kafka/keytabs/kafka_broker1.keytab"
          principal="kafkabroker1/kafka.eigenroute.com@EIGENROUTE.COM";
  };
  </code></pre>
  
  <p>This is the zookeeper file in <code>/etc/init.d</code>:</p>
  
  <pre><code>#!/bin/bash
  export ZOOCFGDIR="/etc/zookeeper/conf/"
  export SERVER_JVMFLAGS="-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper.jaas -Djava.security.krb5.conf=/etc/krb5.conf"
  echo "$@"
  /usr/share/java/zookeeper-3.4.10/bin/zkServer.sh $@ /etc/zookeeper/conf/zoo.cfg
  </code></pre>
  
  <p>Here is my zookeeper configuration file <code>zoo.cfg</code>:</p>
  
  <pre><code>tickTime=2000
  initLimit=10
  syncLimit=5
  dataDir=/var/lib/zookeeper
  clientPort=2181
  authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
  requireClientAuthScheme=sasl
  jaasLoginRenew=3600000
  </code></pre>
  
  <p>I have enabled <code>useTicketCache=true</code> in the zookeeper.jaas file (does it belong there?):</p>
  
  <pre><code>Server {
          com.sun.security.auth.module.Krb5LoginModule required
          useTicketCache=true
          refreshKrb5Config=true
          useKeyTab=true
          keyTab="/home/kafka/keytabs/zookeeper.keytab"
          storeKey=true
          principal="zookeeper/zookeeper.eigenroute.com@EIGENROUTE.COM";
  };
  </code></pre>
  
  <p>Finally, the keytab files are all readable:</p>
  
  <pre><code>$ ll /home/kafka/keytabs/
  total 24
  drwxr-xr-x  2 sjamal sjamal 4096 Dec 12 11:32 .
  drwxr-xr-x 10 kafka  kafka  4096 Dec 12 11:57 ..
  -rw-r--r--  1 root   root    366 Dec 12 11:24 kafka_broker1.keytab
  -rw-r--r--  1 root   root    426 Dec 12 11:31 testkafkaconsumer1.keytab
  -rw-r--r--  1 root   root    426 Dec 12 11:31 testkafkaproducer1.keytab
  -rw-r--r--  1 root   root    370 Dec 12 11:32 zookeeper.keytab
  </code></pre>
  
  <p>I have consulted the following resources:</p>
  
  <ul>
  <li><a href="https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/" rel="noreferrer">apache-kafka-security-authorization-authentication-encryption</a>. 
  <a href="https://stackoverflow.com/questions/43469962/kafka-sasl-zookeeper-authentication">Kafka SASL zookeeper authentication</a> (this says to add the <code>zookeeper.set.acl=true</code> to server.properties, but this has no effect on the issue I am experiencing) </li>
  <li><a href="https://kafka.apache.org/documentation/#configuration" rel="noreferrer">https://kafka.apache.org/documentation/#configuration</a> (the official documentation, which could use improvement in some areas)</li>
  <li><a href="https://coheigea.blogspot.ca/2017/05/securing-apache-kafka-with-kerberos.html" rel="noreferrer">https://coheigea.blogspot.ca/2017/05/securing-apache-kafka-with-kerberos.html</a> (a set of instructions that I initially started following)</li>
  </ul>
  
  <p>Can someone suggest what the problem might be, and how to fix this? Thanks!</p>
  
  <p>UPDATE: I ran <code>netstat -tulnp | grep 2181</code> and <code>ps aux | grep zookeeper</code>, the output below shows that Zookeeper is running and listening on port 2181:</p>
  
  <pre><code>root@devel-2:~# netstat -tulnp | grep 2181
  tcp6       0      0 :::2181                 :::*                    LISTEN      3366/java
  root@devel-2:~# ps aux | grep zookeeper
  root      3366  0.0  0.6 3474796 26000 ?       Sl   Dec16   1:43 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/share/java/zookeeper-3.4.10/bin/../build/classes:/usr/share/java/zookeeper-3.4.10/bin/../build/lib/*.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/slf4j-api-1.6.1.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/netty-3.10.5.Final.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/log4j-1.2.16.jar:/usr/share/java/zookeeper-3.4.10/bin/../lib/jline-0.9.94.jar:/usr/share/java/zookeeper-3.4.10/bin/../zookeeper-3.4.10.jar:/usr/share/java/zookeeper-3.4.10/bin/../src/java/lib/*.jar:/etc/zookeeper/conf/: -Dsun.security.krb5.debug=true -Dlog4j.configuration=file:/etc/zookeeper/conf/log4j.properties -Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper.jaas -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/etc/zookeeper/conf/jaas.conf -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg
  </code></pre>
  
  <p>UPDATE: The software versions are:</p>
  
  <ul>
  <li>Kerberos 5 version 1.12.1</li>
  <li>Zookeeper 3.4.10</li>
  <li>Kafka 0.11.0.0</li>
  </ul>
  
  <p>UPDATE: Below is the content of my <code>/etc/krb5.conf</code> file:</p>
  
  <pre><code>[libdefaults]
          default_realm = EIGENROUTE.COM
  
  # The following krb5.conf variables are only for MIT Kerberos.
          krb4_config = /etc/krb.conf
          krb4_realms = /etc/krb.realms
          kdc_timesync = 1
          ccache_type = 4
          forwardable = true
          proxiable = true
  # The following libdefaults parameters are only for Heimdal Kerberos.
          v4_instance_resolve = false
          v4_name_convert = {
                  host = {
                          rcmd = host
                          ftp = ftp
                  }
                  plain = {
                          something = something-else
                  }
          }
          fcc-mit-ticketflags = true
  
  [realms]
          EIGENROUTE.COM = {
                  kdc = krb.eigenroute.com
                  admin_server = krb.eigenroute.com
                  default_domain = eigenroute.com
          }
          ATHENA.MIT.EDU = {
                  kdc = kerberos.mit.edu:88
                  kdc = kerberos-1.mit.edu:88
                  kdc = kerberos-2.mit.edu:88
                  admin_server = kerberos.mit.edu
                  default_domain = mit.edu
          }
          MEDIA-LAB.MIT.EDU = {
                  kdc = kerberos.media.mit.edu
                  admin_server = kerberos.media.mit.edu
          }
          ZONE.MIT.EDU = {
                  kdc = casio.mit.edu
                  kdc = seiko.mit.edu
                  admin_server = casio.mit.edu
          }
          MOOF.MIT.EDU = {
                  kdc = three-headed-dogcow.mit.edu:88
                  kdc = three-headed-dogcow-1.mit.edu:88
                  admin_server = three-headed-dogcow.mit.edu
          }
          CSAIL.MIT.EDU = {
                  kdc = kerberos-1.csail.mit.edu
                  kdc = kerberos-2.csail.mit.edu
                  admin_server = kerberos.csail.mit.edu
                  default_domain = csail.mit.edu
                  krb524_server = krb524.csail.mit.edu
          }
          IHTFP.ORG = {
                  kdc = kerberos.ihtfp.org
                  admin_server = kerberos.ihtfp.org
          }
          GNU.ORG = {
                  kdc = kerberos.gnu.org
                  kdc = kerberos-2.gnu.org
                  kdc = kerberos-3.gnu.org
                  admin_server = kerberos.gnu.org
          }
          1TS.ORG = {
                  kdc = kerberos.1ts.org
                  admin_server = kerberos.1ts.org
          }
          GRATUITOUS.ORG = {
                  kdc = kerberos.gratuitous.org
                  admin_server = kerberos.gratuitous.org
          }
          DOOMCOM.ORG = {
                  kdc = kerberos.doomcom.org
                  admin_server = kerberos.doomcom.org
          }
          ANDREW.CMU.EDU = {
                  kdc = kerberos.andrew.cmu.edu
                  kdc = kerberos2.andrew.cmu.edu
                  kdc = kerberos3.andrew.cmu.edu
                  admin_server = kerberos.andrew.cmu.edu
                  default_domain = andrew.cmu.edu
          }
          CS.CMU.EDU = {
                  kdc = kerberos.cs.cmu.edu
                  kdc = kerberos-2.srv.cs.cmu.edu
                  admin_server = kerberos.cs.cmu.edu
          }
          DEMENTIA.ORG = {
                  kdc = kerberos.dementix.org
                  kdc = kerberos2.dementix.org
                  admin_server = kerberos.dementix.org
          }
          stanford.edu = {
                  kdc = krb5auth1.stanford.edu
                  kdc = krb5auth2.stanford.edu
                  kdc = krb5auth3.stanford.edu
                  master_kdc = krb5auth1.stanford.edu
                  admin_server = krb5-admin.stanford.edu
                  default_domain = stanford.edu
          }
          UTORONTO.CA = {
                  kdc = kerberos1.utoronto.ca
                  kdc = kerberos2.utoronto.ca
                  kdc = kerberos3.utoronto.ca
                  admin_server = kerberos1.utoronto.ca
                  default_domain = utoronto.ca
          }
  
  [domain_realm]
          .mit.edu = ATHENA.MIT.EDU
          mit.edu = ATHENA.MIT.EDU
          .media.mit.edu = MEDIA-LAB.MIT.EDU
          media.mit.edu = MEDIA-LAB.MIT.EDU
          .csail.mit.edu = CSAIL.MIT.EDU
          csail.mit.edu = CSAIL.MIT.EDU
          .whoi.edu = ATHENA.MIT.EDU
          whoi.edu = ATHENA.MIT.EDU
          .stanford.edu = stanford.edu
          .slac.stanford.edu = SLAC.STANFORD.EDU
          .toronto.edu = UTORONTO.CA
          .utoronto.ca = UTORONTO.CA
          .eigenroute.com = EIGENROUTE.COM
          eigenroute.com = EIGENROUTE.COM
  
  [login]
          krb4_convert = true
          krb4_get_tickets = false
  [logging]
         kdc = FILE:/var/log/kerberos/krb5kdc.log
    admin_server = FILE:/var/log/kerberos/kadmin.log
     default = FILE:/var/log/kerberos/krb5lib.log
  </code></pre>