title: Python Pandas: Error tokenizing data. C error: EOF inside string starting when reading 1GB CSV file
body: <p>I'm reading a 1 GB CSV file in chunks of 10,000 rows. The file has 1106012 rows and 171 columns, other smaller sized file does not show any error and finish off successfully but when i read this 1 GB file it shows error every time on exactly line number 1106011 which is a second last line of file, i can manually remove that line but that is not the solution because i have hundreds of other file of that same size and i cannot fix all the lines manually. can anyone help me with that please.</p>
  
  <pre><code>def extract_csv_to_sql(input_file_name, header_row, size_of_chunk, eachRow):
  
          df = pd.read_csv(input_file_name,
                           header=None,
                           nrows=size_of_chunk,
                           skiprows=eachRow,
                           low_memory=False,
                           error_bad_lines=False,
                           sep=',')
                           # engine='python'
                           # quoting=csv.QUOTE_NONE
                           # encoding='utf-8'
  
          df.columns = header_row
          df = df.drop_duplicates(keep='first')
          df = df.apply(lambda x: x.astype(str).str.lower())
  
          return df
  </code></pre>
  
  <p>I'm then calling this function within a loop and works just fine.</p>
  
  <pre><code>huge_chunk_return = extract_csv_to_sql(huge_input_filename, huge_header_row, the_size_of_chunk_H, each_Row_H)
  </code></pre>
  
  <p>I read this <a href="https://stackoverflow.com/questions/18016037/pandas-parsererror-eof-character-when-reading-multiple-csv-files-to-hdf5">Pandas ParserError EOF character when reading multiple csv files to HDF5</a>, this <a href="https://stackoverflow.com/questions/27158922/read-csv-eof-character-in-string-cause-parsing-issue">read_csv() &amp; EOF character in string cause parsing issue</a> and this <a href="https://github.com/pandas-dev/pandas/issues/11654" rel="nofollow noreferrer">https://github.com/pandas-dev/pandas/issues/11654</a> and many more and tried to include read_csv parameter such as </p>
  
  <blockquote>
    <p>engine='python'</p>
    
    <p>quoting=csv.QUOTE_NONE // Hangs and even the python shell, don't know why</p>
    
    <p>encoding='utf-8'</p>
  </blockquote>
  
  <p>but none of it worked, its still throwing the following error</p>
  
  <p><strong>Error:</strong></p>
  
  <pre><code>Traceback (most recent call last):
    File "C:\Users\WCan\Desktop\wcan_new_python\pandas_test_3.py", line 115, in &lt;module&gt;
      huge_chunk_return = extract_csv_to_sql(huge_input_filename, huge_header_row, the_size_of_chunk_H, each_Row_H)
    File "C:\Users\WCan\Desktop\wcan_new_python\pandas_test_3.py", line 24, in extract_csv_to_sql
      sep=',')
    File "C:\Users\WCan\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 655, in parser_f
      return _read(filepath_or_buffer, kwds)
    File "C:\Users\WCan\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 411, in _read
      data = parser.read(nrows)
    File "C:\Users\WCan\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 1005, in read
      ret = self._engine.read(nrows)
    File "C:\Users\WCan\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\io\parsers.py", line 1748, in read
      data = self._reader.read(nrows)
    File "pandas\_libs\parsers.pyx", line 893, in pandas._libs.parsers.TextReader.read (pandas\_libs\parsers.c:10885)
    File "pandas\_libs\parsers.pyx", line 966, in pandas._libs.parsers.TextReader._read_rows (pandas\_libs\parsers.c:11884)
    File "pandas\_libs\parsers.pyx", line 953, in pandas._libs.parsers.TextReader._tokenize_rows (pandas\_libs\parsers.c:11755)
    File "pandas\_libs\parsers.pyx", line 2184, in pandas._libs.parsers.raise_parser_error (pandas\_libs\parsers.c:28765)
  pandas.errors.ParserError: Error tokenizing data. C error: EOF inside string starting at line 1106011
  &gt;&gt;&gt; 
  </code></pre>