title: Reading null values from CSV in Spark with a schema defining nullable = false doesn't behave as expected
body: <p>When loading a CSV file defining a schema where some fields are marked with <code>nullable = false</code>, I would expect those rows containing null values for the specified columns to be dropped or filtered out of the dataset when also defining a mode of <code>DROPMALFORMED</code>. This may be a misunderstanding on my part on exactly what is considered malformed, but in any case, I'm confused as to how the code continues to work when the schema explicitly defines certain fields as not accepting null values.</p>
  
  <p>Looking at the databricks docs for the CSV reader (I know that this functionality has now been rolled into Apache Spark directly, but I can't find documentation for it), it implies that the schema should be taken into account when reading values.</p>
  
  <p><a href="https://github.com/databricks/spark-csv#features" rel="nofollow noreferrer">https://github.com/databricks/spark-csv#features</a></p>
  
  <blockquote>
    <p>DROPMALFORMED: drops lines which have fewer or more tokens than expected <strong>or tokens which do not match the schema</strong></p>
  </blockquote>
  
  <p>Example:</p>
  
  <p>CSV file:</p>
  
  <pre><code>value1,value2,,value4
  </code></pre>
  
  <p>Spark code (in Scala):</p>
  
  <pre><code>val spark = SparkSession
    .builder()
    .appName("example")
    .master("local")
    .getOrCreate()
  
  spark.read.
    .format("csv")
    .schema(StructType(
      StructField("col1", StringType),
      StructField("col2", StringType),
      StructField("col3", StringType, nullable = false),
      StructField("col4", StringType)))
    .option("header", false)
    .option("mode", "DROPMALFORMED")
    .load("path/to/file.csv")
  </code></pre>
  
  <p>The above code will still include the row containing the null value for <code>col3</code>, and in fact, if I want to filter out records with a null value, I have to do the following:</p>
  
  <pre><code>  .filter(row =&gt; !row.isNullAt(row.fieldIndex("col3")))
  </code></pre>
  
  <p>So my questions are:  </p>
  
  <p>1) Was my assumption that DROPMALFORMED mode would drop data not conforming to the schema an invalid one?<br>
  2) Have I done something wrong in the way I'm loading the CSV which has resulted in the unexpected behaviour above (or perhaps there's a better cleaner way of doing what I want)?</p>
  
  <p>My code looks almost identical to the examples on databricks' documentation, and other examples of loading CSVs using Spark found online.</p>
  
  <p>I'm using Spark <code>2.3.1</code> with Scala <code>2.11.8</code>.</p>
  
  <p>[edit]<br>
  I have raised this issue on the Apache Spark's JIRA: <a href="https://issues.apache.org/jira/browse/SPARK-25545" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/SPARK-25545</a></p>