title: Read elasticsearch index using Pyspark
body: <p><strong>I am trying to read elasticsearch index using Pyspark (v1.6.3), but getting following error</strong></p>
  
  <p>I am using following snippet to read/load the index</p>
  
  <pre><code>    es_reader = sql_context.read.format("org.elasticsearch.spark.sql")
                  .option("es.nodes", "x.x.x.x,y.y.y.y,z.z.z.z")
                  .option("es.port", "9200").option("es.net.ssl","true")
                  .option("es.net.http.auth.user", "*****")
                  .option("es.net.http.auth.pass", "*****")
      sde_df = es_reader.load("index_name/doc_type")
  </code></pre>
  
  <p><strong>Error:</strong></p>
  
  <pre><code>22_0452/container_1547624497922_0452_02_000001/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
    File "/mnt/yarn/usercache/root/appcache/application_1547624497922_0452/container_1547624497922_0452_02_000001/pyspark.zip/pyspark/sql/utils.py", line 45, in deco
    File "/mnt/yarn/usercache/root/appcache/application_1547624497922_0452/container_1547624497922_0452_02_000001/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
  py4j.protocol.Py4JJavaError: An error occurred while calling o72.load.
  : java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/StreamSinkProvider
  
  Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.StreamSinkProvider
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 49 more
  </code></pre>
  
  <p><strong>Environment:</strong></p>
  
  <p>Spark v1.6.3</p>
  
  <p>Elasticsearch 5.4.3</p>