title: Spark is loading some rows into one column?
body: <p>I currently have a table that looks like this</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>keyword</th>
  <th>id</th>
  <th>date</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>apple</td>
  <td>A1</td>
  <td>2021-01-01</td>
  </tr>
  <tr>
  <td>apple</td>
  <td>A2</td>
  <td>2021-01-02</td>
  </tr>
  <tr>
  <td>apple</td>
  <td>A3</td>
  <td>2021-01-02</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>After I save this table to S3 in TSV uncompressed, I check the distinct dates, which are correct (2020-01-01, 2020-01-2).</p>
  <p>However, when I load the files from S3, for approximately .002% of the billion rows, the table now looks like this. The column values are all put into the first column, and the values are not guaranteed to be correct.</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>keyword</th>
  <th>id</th>
  <th>date</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>apple  A1 2021-01-02</td>
  <td>null</td>
  <td>null</td>
  </tr>
  <tr>
  <td>apple A2 2021-01-01</td>
  <td>null</td>
  <td>null</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>while the vast majority of the table will look as intended. At the end, the distinct dates now are (2021-01-01, 2021-01-02, and null)</p>
  <p>There is something strange going on with my load function, but I am not sure how to identify the root cause of this issue</p>
  <p>My load function:</p>
  <pre><code>def load(s3Path: String): DataFrame = {
      sparkSession.read
        .format(&quot;com.databricks.spark.csv&quot;)
        .option(&quot;header&quot;, true)
        .option(&quot;maxCharsPerColumn&quot;, MAX_CHARS_PER_COLUMN)
        .option(&quot;delimiter&quot;, &quot;\t&quot;)
        .option(&quot;inferSchema&quot;, true)
        .load(s3Path)
    }
  </code></pre>
  <p>I have inferSchema set to true. Is that a possible root cause?</p>
  <p>EDIT: I found my root cause.
  Some keywords have a dangling quote, so it looks like &quot;apple, and when Spark loads it, the row starting with an unclosed quote will be put into the keyword column.</p>