title: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities. There is no version issue
body: <p>I am trying to read data from aws s3 into dataset/rdd in Java. I am running Spark code in Java on IntelliJ, so added Hadoop dependencies as well in pom.xml</p>
  
  <p>Below is my code and pom.xml file.</p>
  
  <pre><code>    import org.apache.spark.api.java.JavaRDD;
      import org.apache.spark.sql.SparkSession;
      import org.apache.spark.api.java.JavaSparkContext;
  
      public class SparkJava {
  
          public static void main(String[] args){
  
              SparkSession spark  = SparkSession
                      .builder()
                      .master("local")
                      .config("spark.hadoop.fs.s3a.impl","org.apache.hadoop.fs.s3a.S3AFileSystem")                  
                      .config("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", "2")
                      .config("fs.s3n.awsAccessKeyId", AWS_KEY)
                      .config("fs.s3n.awsSecretAccessKey", AWS_SECRET_KEY)
                      .getOrCreate();
  
              JavaSparkContext sc = new JavaSparkContext(spark.sparkContext());
              String input_path = "s3a://bucket/2018/07/28";
              JavaRDD&lt;String&gt; s3aRdd = sc.textFile(input_path);
              long count  = s3aRdd.count();  // THIS IS CAUSING EXCEPTION
              System.out.print(count);
              System.out.print("Finished");         
          }
      }
  </code></pre>
  
  <p>Here are the dependencies from pom.xml</p>
  
  <pre><code>&lt;dependencies&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-core_2.11&lt;/artifactId&gt;
          &lt;version&gt;2.3.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
          &lt;version&gt;2.3.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
          &lt;artifactId&gt;hadoop-aws&lt;/artifactId&gt;
          &lt;version&gt;3.1.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
          &lt;artifactId&gt;hadoop-hdfs&lt;/artifactId&gt;
          &lt;version&gt;3.1.1&lt;/version&gt;
      &lt;/dependency&gt;
  &lt;/dependencies&gt;
  </code></pre>
  
  <p>There is no version issue in this case as mentioned in this question: <a href="https://stackoverflow.com/questions/52310416/noclassdeffounderror-org-apache-hadoop-fs-streamcapabilities-while-reading-s3-d">NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities while reading s3 Data with spark</a></p>