title: PCF Kafka with Kerberos authentication - Running Locally but Failing in PCF
body: <p>I am facing issue when Connecting to Kafka through SpringBoot using Kerberos Authentication. I am using the custom Kafka connection manager with following details -</p>
  <pre><code>          bootstrap-servers-sasl: node1:9094, node2:9094, node3:9094
            protocol: SASL_SSL
            mechanism: GSSAPI
            kerberos:
              service:
                name: kfkusr
            jaas: 
              config: &quot;com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\&quot;#keytab-name#\&quot; principal=\&quot;abc/node2@domain.NET\&quot;;&quot;
  </code></pre>
  <p>where <code>#keytab-name#</code> will be replaced at runtime with following values as -</p>
  <p>My Local PC - <code>C:/Users/MyPC/AppData/Local/Temp/abc.node2_d2254866264751402128.keytab</code></p>
  <p>PCF - <code>/home/vcap/tmp/abc.node2_d2215947326380395062.keytab</code></p>
  <p>Locally the application is running fine and messages are going to Kafka. But when running on PCF fails with following exception -</p>
  <pre><code>2019-08-09T14:40:46.481-05:00 [APP/PROC/WEB/0] [OUT] WARN [9f-3868cbe47d81] org.apache.kafka.clients.NetworkClient o.a.k.c.NetworkClient.processDisconnection(NetworkClient.java:585) - ||||||||||||||Connection to node -1 terminated during authentication. This may indicate that authentication failed due to invalid credentials.
  ...
  ...
  Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms
  ...
  ...
  Exception thrown when sending a message with key='null' and payload='&lt;my payload&gt;' to topic &lt;test_topic&gt; :: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
  </code></pre>
  <p><strong>UPDATE 1 -</strong></p>
  <p>After adding the krb5.conf file and it has default realm</p>
  <pre><code>[libdefaults]
    default_realm = mydomain.NET
  </code></pre>
  <p>the authentication error disappeared but still have following error</p>
  <pre><code>Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms
  </code></pre>
  <p>Once Debug enabled for <code>org.apache.kafka: DEBUG</code> is showing new error as -</p>
  <pre><code>2019-08-14T09:49:51.947-05:00 [APP/PROC/WEB/0] [OUT] DEBUG [d3-5b28248c661c] org.apache.kafka.clients.NetworkClient o.a.k.c.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:907) - ||||||||||||||Initialize connection to node node1:9094 (id: -1 rack: null) for sending metadata request
  2019-08-14T09:49:51.947-05:00 [APP/PROC/WEB/0] [OUT] DEBUG [d3-5b28248c661c] org.apache.kafka.clients.NetworkClient o.a.k.c.NetworkClient.initiateConnect(NetworkClient.java:762) - ||||||||||||||Initiating connection to node node1:9094 (id: -1 rack: null)
  2019-08-14T09:49:51.948-05:00 [APP/PROC/WEB/0] [OUT] DEBUG [d3-5b28248c661c] o.a.k.c.s.a.SaslClientAuthenticator o.a.k.c.s.a.SaslClientAuthenticator.setSaslState(SaslClientAuthenticator.java:209) - ||||||||||||||Set SASL client state to SEND_HANDSHAKE_REQUEST
  2019-08-14T09:49:51.948-05:00 [APP/PROC/WEB/0] [OUT] DEBUG [d3-5b28248c661c] o.a.k.c.s.a.SaslClientAuthenticator o.a.k.c.s.a.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:134) - ||||||||||||||Creating SaslClient: client=abc/node1@domain.net;service=kfkusr;serviceHostname=node1;mechs=[GSSAPI]
  2019-08-14T09:49:51.949-05:00 [APP/PROC/WEB/0] [OUT] INFO [d3-5b28248c661c] o.a.k.common.network.SaslChannelBuilder o.a.k.c.n.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:119) - ||||||||||||||Failed to create channel due to : org.apache.kafka.common.KafkaException: Failed to configure SaslClientAuthenticator at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.configure(SaslClientAuthenticator.java:125) at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:116) at org.apache.kafka.common.network.Selector.connect(Selector.java:203) at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:764) at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:60) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:908) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:819) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:431) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.kafka.common.KafkaException: Failed to create SaslClient with mechanism GSSAPI at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:140) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.configure(SaslClientAuthenticator.java:123) ... 10 common frames omittedCaused by: javax.security.sasl.SaslException: Failure to initialize security context at com.sun.security.sasl.gsskerb.GssKrb5Client.&lt;init&gt;(GssKrb5Client.java:149) at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:63) at javax.security.sasl.Sasl.createSaslClient(Sasl.java:384) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:136) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:131) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:131) ... 11 common frames omittedCaused by: org.ietf.jgss.GSSException: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm) at sun.security.jgss.krb5.Krb5NameElement.getInstance(Krb5NameElement.java:129) at sun.security.jgss.krb5.Krb5MechFactory.getNameElement(Krb5MechFactory.java:95) at sun.security.jgss.GSSManagerImpl.getNameElement(GSSManagerImpl.java:203) at sun.security.jgss.GSSNameImpl.getElement(GSSNameImpl.java:477) at sun.security.jgss.GSSNameImpl.init(GSSNameImpl.java:201) at sun.security.jgss.GSSNameImpl.&lt;init&gt;(GSSNameImpl.java:170) at sun.security.jgss.GSSManagerImpl.createName(GSSManagerImpl.java:138) at com.sun.security.sasl.gsskerb.GssKrb5Client.&lt;init&gt;(GssKrb5Client.java:107) ... 18 common frames omitted
  2019-08-14T09:49:51.949-05:00 [APP/PROC/WEB/0] [OUT] DEBUG [d3-5b28248c661c] org.apache.kafka.clients.NetworkClient o.a.k.c.NetworkClient.initiateConnect(NetworkClient.java:773) - ||||||||||||||Error connecting to node abcNode:9094 (id: -1 rack: null): java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed] at org.apache.kafka.common.network.Selector.connect(Selector.java:210) at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:764) at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:60) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:908) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:819) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:431) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:224) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:162) at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to configure SaslClientAuthenticator at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:120) at org.apache.kafka.common.network.Selector.connect(Selector.java:203) ... 8 common frames omittedCaused by: org.apache.kafka.common.KafkaException: Failed to configure SaslClientAuthenticator at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.configure(SaslClientAuthenticator.java:125) at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:116) ... 9 common frames omittedCaused by: org.apache.kafka.common.KafkaException: Failed to create SaslClient with mechanism GSSAPI at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:140) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.configure(SaslClientAuthenticator.java:123) ... 10 common frames omittedCaused by: javax.security.sasl.SaslException: Failure to initialize security context at com.sun.security.sasl.gsskerb.GssKrb5Client.&lt;init&gt;(GssKrb5Client.java:149) at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslClient(FactoryImpl.java:63) at javax.security.sasl.Sasl.createSaslClient(Sasl.java:384) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:136) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$1.run(SaslClientAuthenticator.java:131) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslClient(SaslClientAuthenticator.java:131) ... 11 common frames omittedCaused by: org.ietf.jgss.GSSException: Invalid name provided (Mechanism level: KrbException: Cannot locate default realm) at sun.security.jgss.krb5.Krb5NameElement.getInstance(Krb5NameElement.java:129) at sun.security.jgss.krb5.Krb5MechFactory.getNameElement(Krb5MechFactory.java:95) at sun.security.jgss.GSSManagerImpl.getNameElement(GSSManagerImpl.java:203) at sun.security.jgss.GSSNameImpl.getElement(GSSNameImpl.java:477) at sun.security.jgss.GSSNameImpl.init(GSSNameImpl.java:201) at sun.security.jgss.GSSNameImpl.&lt;init&gt;(GSSNameImpl.java:170) at sun.security.jgss.GSSManagerImpl.createName(GSSManagerImpl.java:138) at com.sun.security.sasl.gsskerb.GssKrb5Client.&lt;init&gt;(GssKrb5Client.java:107) ... 18 common frames omitted
  
  </code></pre>