title: Spark SQL - loading csv/psv files with some malformed records
body: <p>We are loading hierarchies of directories of files with Spark and converting them to Parquet. There are tens of gigabytes in hundreds of pipe-separated files. Some are pretty big themselves.</p>
  
  <p>Every, say, 100th file has a row or two that has an extra delimiter that makes the whole process (or the file) abort.</p>
  
  <p>We are loading using:</p>
  
  <pre><code>sqlContext.read
          .format("com.databricks.spark.csv")
          .option("header", format("header"))
          .option("delimiter", format("delimeter"))
          .option("quote", format("quote"))
          .option("escape", format("escape"))
          .option("charset", "UTF-8")
          // Column types are unnecessary for our current use cases.
          //.option("inferschema", "true")
          .load(glob)
  </code></pre>
  
  <p>Is there any extension or a event handling mechanism with Spark that we could attach to the logic that reads rows, that, if the malformed row is encountered, just skips the row instead of failing the process on it?</p>
  
  <p>(We are planning to do more pre-processing, but this would be the most immediate and critical fix.) </p>