title: Problems starting new Kafka Connect SSL-based connector
body: <p>Im trying to setup a new ElasticSearchSink-job on our KafkaConnect cluster. The cluster has been working smoothly for a couple of months with a SASL-SSL secured connection to Kafka and HTTPS to an Elastic-instance on host A. <br/><br/>The KC-Cluster normally run in Kubernetes but for testing purposes I also run it locally using docker(image based on Confluent's KC-image v6.0.0), the Kafka resides in a test-environment and the job is started using REST-calls.</p>
  <p>The docker-composed-file used for running it localy looks like this</p>
  <pre><code>version: '3.7'
  services:
    connect:
      build:
        dockerfile: Dockerfile.local
        context: ./
      container_name: kafka-connect
      ports:
        - &quot;8083:8083&quot;
      environment:
        KAFKA_OPTS: -Djava.security.krb5.conf=/&lt;path-to&gt;/secrets/krb5.conf 
                    -Djava.security.auth.login.config=/&lt;path-to&gt;/rest-basicauth-jaas.conf
        CONNECT_BOOTSTRAP_SERVERS: &lt;KAFKA-INSTANCE-1&gt;:2181,&lt;KAFKA-INSTANCE-2&gt;:2181,&lt;KAFKA-INSTANCE-3&gt;:2181
        CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect
        CONNECT_REST_PORT: 8083
        CONNECT_REST_EXTENSION_CLASSES: org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension
        CONNECT_GROUP_ID: &lt;kc-group&gt;
        CONNECT_CONFIG_STORAGE_TOPIC: service-assurance.test.internal.connect.configs
        CONNECT_OFFSET_STORAGE_TOPIC: service-assurance.test.internal.connect.offsets
        CONNECT_STATUS_STORAGE_TOPIC: service-assurance.test.internal.connect.status
        CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
        CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
        CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
        CONNECT_KEY_CONVERTER: org.apache.kafka.connect.converters.IntegerConverter
        CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
        CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
        CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
        CONNECT_ZOOKEEPER_CONNECT: &lt;KAFKA-INSTANCE-1&gt;:2181,&lt;KAFKA-INSTANCE-2&gt;:2181,&lt;KAFKA-INSTANCE-3&gt;:2181
        CONNECT_SECURITY_PROTOCOL: SASL_SSL
        CONNECT_SASL_KERBEROS_SERVICE_NAME: &quot;kafka&quot;
        CONNECT_SASL_JAAS_CONFIG: com.sun.security.auth.module.Krb5LoginModule required \
                                  useKeyTab=true \
                                  storeKey=true \
                                  keyTab=&quot;/&lt;path-to&gt;/kafka-connect.keytab&quot; \
                                  principal=&quot;&lt;AD-USER&gt;&quot;;
        CONNECT_SASL_MECHANISM: GSSAPI
        CONNECT_SSL_TRUSTSTORE_LOCATION: &quot;/&lt;path-to&gt;/truststore.jks&quot;
        CONNECT_SSL_TRUSTSTORE_PASSWORD: &lt;pwd&gt;
        CONNECT_CONSUMER_SECURITY_PROTOCOL: SASL_SSL
        CONNECT_CONSUMER_SASL_KERBEROS_SERVICE_NAME: &quot;kafka&quot;
        CONNECT_CONSUMER_SASL_JAAS_CONFIG: com.sun.security.auth.module.Krb5LoginModule required \
                                  useKeyTab=true \
                                  storeKey=true \
                                  keyTab=&quot;/&lt;path-to&gt;/kafka-connect.keytab&quot; \
                                  principal=&quot;&lt;AD-USER&gt;&quot;;
        CONNECT_CONSUMER_SASL_MECHANISM: GSSAPI
        CONNECT_CONSUMER_SSL_TRUSTSTORE_LOCATION: &quot;/&lt;path-to&gt;/truststore.jks&quot;
        CONNECT_CONSUMER_SSL_TRUSTSTORE_PASSWORD: &lt;pwd&gt;
        CONNECT_PLUGIN_PATH: &quot;/usr/share/java,/etc/kafka-connect/jars&quot;
  </code></pre>
  <p>With a similar kuberneted-configuration.</p>
  <p>The connector is started using something like:</p>
  <pre><code>curl  -X POST -H &quot;Content-Type: application/json&quot; --data '{
    &quot;name&quot;: &quot;connector-name&quot;,
    &quot;config&quot;: {
      &quot;connector.class&quot;: &quot;io.confluent.connect.elasticsearch.ElasticsearchSinkConnector&quot;,
      &quot;tasks.max&quot;: 2,
      &quot;batch.size&quot;: 200,
      &quot;max.buffered.records&quot;: 1500,
      &quot;flush.timeout.ms&quot;: 120000,
      &quot;topics&quot;: &quot;topic.connector&quot;,
      &quot;auto.create.indices.at.start&quot;: false,
      &quot;key.ignore&quot;: true,
      &quot;value.converter.schemas.enable&quot;: false,
      &quot;key.converter&quot;: &quot;org.apache.kafka.connect.converters.ByteArrayConverter&quot;,
      &quot;schema.ignore&quot;: true,
      &quot;value.converter&quot;: &quot;org.apache.kafka.connect.json.JsonConverter&quot;,
      &quot;behavior.on.malformed.documents&quot; : &quot;ignore&quot;,
      &quot;behavior.on.null.values&quot;: &quot;ignore&quot;,
      &quot;connection.url&quot;: &quot;https://&lt;elastic-host&gt;&quot;,
      &quot;connection.username&quot;: &quot;&lt;user&gt;&quot;,
      &quot;connection.password&quot;: &quot;&lt;pwd&gt;&quot;,
      &quot;type.name&quot;: &quot;_doc&quot;
    }
  }' &lt;host&gt;/connectors/
  </code></pre>
  <p>Now, Ive been tasked to setup yet another connector, this time hosted on host B. The problem that I am experiencing is the infamous:</p>
  <pre><code>    sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  </code></pre>
  <p>I have modified the working truststore to include both the CA-Root-certificate for host B aswell. I belive the truststore is working as I am able to use it from a java code-snippet (actually found on a Atlassian page, SSLPoke.class) to connect to both A and B succesfully.</p>
  <p>The connectors connecting to host A still work with the newly updated truststore but not the connector connecting to host B.</p>
  <p>I have scanned to internet for clues on how to solve this and come across  suggestions to explicitly add:</p>
  <pre><code>&quot;elastic.https.ssl.truststore.location&quot;: &quot;/&lt;pathto&gt;/truststore.jks&quot;,
  &quot;elastic.https.ssl.truststore.password&quot;: &quot;&lt;pwd&gt;&quot;,
  </code></pre>
  <p>To the connector-configuration. Some other page suggested adding the truststore to the KC-configuration KAFKA_OPTS as such:</p>
  <pre><code>  KAFKA_OPTS: -Djava.security.krb5.conf=/&lt;path-to&gt;/secrets/krb5.conf 
                -Djava.security.auth.login.config=/&lt;path-to&gt;/rest-basicauth-jaas.conf
                -Djavax.net.ssl.trustStore=/&lt;path-to&gt;/truststore.jks
  </code></pre>
  <p>Following these suggestions I can actually get the connector connecting to host B to start succefully. But now comes the anoying part. With adding the extra param to KAFKA_OPTS my old connectors connecting to A stops working!! - with the exact same error! So now I have a case of either having connectors connecting to A OR connectors connecting to B working, but not at the same time.</p>
  <p>Please, If anyone could give me some pointers or ideas on how to fix this it would be much appreciated, cause this is driving me nuts.</p>