title: Spark - Avro Reads Schema but DataFrame Empty
body: <p>I am using <a href="http://gobblin.readthedocs.io/en/latest/" rel="nofollow">Gobblin</a> to periodically extract relational data from <code>Oracle</code>, convert it to <code>avro</code> and publish it to <code>HDFS</code></p>
  
  <p>My dfs directory structure looks like this</p>
  
  <pre><code>-tables
    |
    -t1
     |
     -2016080712345
      |
      -f1.avro
     |
     -2016070714345
      |
      -f2.avro
  </code></pre>
  
  <p>I am trying to read from it like so:</p>
  
  <pre><code>val sq = sqlContext.read.format("com.databricks.spark.avro")
    .load("/user/username/gobblin/job-output/tables/t1/")
  </code></pre>
  
  <p>When I run <code>printSchema</code> I can see that the schema is interpreted correctly.</p>
  
  <p>However, when I run <code>count</code> or <code>show</code>, the <code>DataFrames</code> are empty. I have verified that the <code>.avro</code> files are not empty by converting it to <code>JSON</code></p>
  
  <pre><code>java -jar avro-tools-1.7.7.jar  tojson --pretty t1/20160706230001_append/part.f1.avro &gt; t1.json
  </code></pre>
  
  <p>I suspect that it may have something to do with the directory structure. Perhaps the Spark avro libraries only look one level down from the root for <code>.avro</code> files. The logs seem to indicate that only the directories under t1 were listed on the driver:</p>
  
  <blockquote>
    <p>16/07/07 10:47:09 INFO avro.AvroRelation: Listing hdfs://myhost.mydomain.com:8020/user/username/gobblin/job-output/tables/t1 on driver</p>
    
    <p>16/07/07 10:47:09 INFO avro.AvroRelation: Listing hdfs://myhost.mydomain.com:8020/user/username/gobblin/job-output/tables/t1/20160706230001_append on driver</p>
  </blockquote>
  
  <p>Has anyone experienced something similar, or know how to get around this? I'd hat to have to point lower than the <code>t1</code> directory because the names are generated by a timestamp. </p>