title: Ignoring corrupted Orc files when reading via Spark
body: <p>I have multiple Orc files in HDFS with the following directory structure:</p>
  <pre class="lang-sh prettyprint-override"><code>orc/
  ├─ data1/
  │  ├─ 00.orc
  │  ├─ 11.orc
  ├─ data2/
  │  ├─ 22.orc
  │  ├─ 33.orc
  </code></pre>
  <p>I am reading these files using Spark:</p>
  <pre class="lang-scala prettyprint-override"><code>spark.sqlContext.read.format(&quot;orc&quot;).load(&quot;/orc/data*/&quot;)
  </code></pre>
  <p>The problem is one of the files is corrupted so I want to skip/ignore that file.</p>
  <p>The only way I see is to get all the Orc files and validate(By reading them) one by one before passing it to Spark. But this way I will be reading the same files twice.</p>
  <p>Is there any way I can avoid reading the files twice? Does Spark provide anything regarding this?</p>