title: How to connect to Elasticsearch by PySpark without cert verification?
body: <p>Is there a way to connect to Elasticsearch (https://IP:9200) by PySpark without certificate verification?</p>
  <p>I didn't see any settings for this: <a href="https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html" rel="nofollow noreferrer">https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html</a></p>
  <p>Curl works fine:</p>
  <pre><code>curl -iv --insecure --user admin:password --noproxy '*' https://xx.xxx.xx.xxx:9200
  *   Trying xx.xxx.xx.xxx:9200...
  * Connected to xx.xxx.xx.xxx (xx.xxx.xx.xxx) port 9200 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * successfully set certificate verify locations:
  *   CAfile: /opt/conda/ssl/cacert.pem
    CApath: none
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  * TLSv1.3 (IN), TLS handshake, Server hello (2):
  * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  * TLSv1.3 (IN), TLS handshake, Certificate (11):
  * TLSv1.3 (IN), TLS handshake, CERT verify (15):
  * TLSv1.3 (IN), TLS handshake, Finished (20):
  * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  * TLSv1.3 (OUT), TLS handshake, Finished (20):
  * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  * ALPN, server did not agree to a protocol
  * Server certificate:
  *  subject: OU=dev-1; CN=dev-1-es-http.es-dev.es.local
  *  start date: Jul  9 00:47:04 2020 GMT
  *  expire date: Jul  9 00:57:04 2021 GMT
  *  issuer: OU=dev-1; CN=dev-1-http
  *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
  * Server auth using Basic with user 'admin'
  &gt; GET / HTTP/1.1
  &gt; Host: xx.xxx.xx.xxx:9200
  &gt; Authorization: Basic abc==
  &gt; User-Agent: curl/7.71.1
  &gt; Accept: */*
  &gt;
  * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  * Mark bundle as not supporting multiuse
  &lt; HTTP/1.1 200 OK
  HTTP/1.1 200 OK
  &lt; content-type: application/json; charset=UTF-8
  content-type: application/json; charset=UTF-8
  &lt; content-length: 544
  content-length: 544
  
  &lt;
  {
    &quot;name&quot; : &quot;dev-1-es-hot-2&quot;,
    &quot;cluster_name&quot; : &quot;dev-1&quot;,
    &quot;cluster_uuid&quot; : &quot;444&quot;,
    &quot;version&quot; : {
      &quot;number&quot; : &quot;7.8.0&quot;,
      &quot;build_flavor&quot; : &quot;default&quot;,
      &quot;build_type&quot; : &quot;docker&quot;,
      &quot;build_hash&quot; : &quot;123&quot;,
      &quot;build_date&quot; : &quot;2020-06-14T19:35:50.234439Z&quot;,
      &quot;build_snapshot&quot; : false,
      &quot;lucene_version&quot; : &quot;8.5.1&quot;,
      &quot;minimum_wire_compatibility_version&quot; : &quot;6.8.0&quot;,
      &quot;minimum_index_compatibility_version&quot; : &quot;6.0.0-beta1&quot;
    },
    &quot;tagline&quot; : &quot;You Know, for Search&quot;
  }
  * Connection #0 to host xx.xxx.xx.xxx left intact
  </code></pre>
  <p>My last try:</p>
  <pre><code>reader = spark.read.format(&quot;org.elasticsearch.spark.sql&quot;) \
      .option(&quot;es.port&quot;, &quot;9200&quot;) \
      .option(&quot;spark.es.net.http.auth.user&quot;, &quot;admin&quot;) \
      .option(&quot;spark.es.net.http.auth.pass&quot;, &quot;password&quot;) \
      .option(&quot;es.net.ssl&quot;, &quot;true&quot;) \
      .option(&quot;es.net.ssl.cert.allow.self.signed&quot;, &quot;true&quot;) \
      .option(&quot;es.net.proxy.https.use.system.props&quot;, &quot;false&quot;) \
      .option(&quot;es.nodes&quot;, &quot;xx.xxx.xxx.xxx&quot;)
  </code></pre>
  <p>Ends with:</p>
  <pre><code>: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
      at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:348)
      at org.elasticsearch.spark.sql.ElasticsearchRelation.cfg$lzycompute(DefaultSource.scala:225)
      at org.elasticsearch.spark.sql.ElasticsearchRelation.cfg(DefaultSource.scala:223)
      at org.elasticsearch.spark.sql.ElasticsearchRelation.lazySchema$lzycompute(DefaultSource.scala:229)
      at org.elasticsearch.spark.sql.ElasticsearchRelation.lazySchema(DefaultSource.scala:229)
      at org.elasticsearch.spark.sql.ElasticsearchRelation.$anonfun$schema$1(DefaultSource.scala:233)
      at scala.Option.getOrElse(Option.scala:189)
      at org.elasticsearch.spark.sql.ElasticsearchRelation.schema(DefaultSource.scala:233)
      at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:448)
      at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
      at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
      at scala.Option.getOrElse(Option.scala:189)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:240)
      at jdk.internal.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:282)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:238)
      at java.base/java.lang.Thread.run(Thread.java:834)
  Caused by: org.elasticsearch.hadoop.rest.EsHadoopTransportException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:128)
      at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:432)
      at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
      at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)
      at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)
      at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:168)
      at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:745)
      at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:338)
      ... 23 more
  Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
      at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
      at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1356)
      at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1231)
      at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1174)
      at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
      at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
      at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
      at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
      at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
      at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1408)
      at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1314)
      at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
      at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:819)
      at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1189)
      at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
      at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
      at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
      at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
      at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
      at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
      at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
      at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
      at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.doExecute(CommonsHttpTransport.java:685)
      at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:664)
      at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:116)
      ... 30 more
  Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
      at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
      at java.base/sun.security.validator.Validator.validate(Validator.java:264)
      at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
      at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:233)
      at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:110)
      at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory$TrustManagerDelegate.checkServerTrusted(SSLSocketFactory.java:76)
      at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1510)
      at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1340)
      ... 54 more
  Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
      at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
      at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
      at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
      ... 62 more
  </code></pre>
  <p>I also trying to provide keystore/truestore without results.</p>