title: Unexpected behavior when reading a 5GB CSV file into pandas dataframe
body: <p>I have a large (5GB) CSV file that I want to load into a <code>pandas.DataFrame</code>. I tried to run the following:</p>
  
  <pre><code>df = pd.read_csv('./original-utf8.csv')
  </code></pre>
  
  <p>But it yielded an error:</p>
  
  <pre><code>CParserError: Error tokenizing data. C error: Expected 92 fields in line 3098834, saw 163
  </code></pre>
  
  <p>I inspected the line <code>3098834 +/- 1</code> manually, and it was OK.</p>
  
  <p>Then, I tried to ignore the offending lines:</p>
  
  <pre><code>df = pd.read_csv('./original-utf8.csv', error_bad_lines=False)
  </code></pre>
  
  <p>This time, I got a 23 warnings of the following style:</p>
  
  <pre><code>b'Skipping line 3098834: expected 92 fields, saw 163\n'
  </code></pre>
  
  <p>Seems like non of them is problematic.</p>
  
  <p>In addition, the number of rows in the yielded <code>DataFrame</code> doesn't match the number of lines in the CSV file (even after taking into account the number of warnings). Seems like <code>pandas</code>(?) drops more lines without saying anything.</p>
  
  <p>Any idea? How can I detect where is the problem?</p>