title: How to solve org.apache.spark.sql.AnalysisException: Path does not exist
body: <p>I use Spark 2.2.0.</p>
  
  <p>I use Spark to process datasets from S3. It worked fine until I decided to use wildcards in order to read data from sub-folders of the folder <code>test</code>.</p>
  
  <pre><code>val path = "s3://data/test"
  val spark = SparkSession
    .builder()
    .appName("Test")
    .config("spark.sql.warehouse.dir", path)
    .enableHiveSupport()
    .getOrCreate()
  
  import spark.implicits._    
  val myData = spark.read.parquet(path + "/*/")
  </code></pre>
  
  <p>I get the following error:</p>
  
  <blockquote>
    <p>17/11/20 18:54:21 ERROR ApplicationMaster: User class threw exception:
    org.apache.spark.sql.AnalysisException: Path does not exist:
    hdfs://ip-111-112-11-65.eu-west-1.compute.internal:8020/user/hdfs/s3/data/test/20171120/*;</p>
  </blockquote>
  
  <p>I execute the above code with the following command:</p>
  
  <pre><code>spark-submit --deploy-mode cluster --driver-memory 10g
  </code></pre>
  
  <p>What I don't understand is why Spark tries to read from HDFS instead of reading from the provided path. The same piece of code works fine with another path like <code>s3://data/test2/mytest.parquet</code>.</p>