title: pandas "pandas.errors.ParserError: Error tokenizing data. C error: Unknown error in IO callback"
body: <p>I'm using pandas to <code>read_csv</code> a 3.8 Gig text file, pipe-delimited, but it errors when reading the file into memory.</p>
  
  <p>Here is the full error thrown out of my <code>read_in_files()</code> function:</p>
  
  <h3>Error:</h3>
  
  <pre><code>Reading in file C:\Users\cdabel\Desktop\_Temp\Master_Extract_Data_Mart_201909240935.txt
  Traceback (most recent call last):
    File "&lt;stdin&gt;", line 10, in &lt;module&gt;
    File "&lt;stdin&gt;", line 7, in read_in_files
    File "c:\python36\lib\site-packages\pandas\io\parsers.py", line 685, in parser_f
      return _read(filepath_or_buffer, kwds)
    File "c:\python36\lib\site-packages\pandas\io\parsers.py", line 463, in _read
      data = parser.read(nrows)
    File "c:\python36\lib\site-packages\pandas\io\parsers.py", line 1154, in read
      ret = self._engine.read(nrows)
    File "c:\python36\lib\site-packages\pandas\io\parsers.py", line 2048, in read
      data = self._reader.read(nrows)
    File "pandas\_libs\parsers.pyx", line 879, in pandas._libs.parsers.TextReader.read
    File "pandas\_libs\parsers.pyx", line 894, in pandas._libs.parsers.TextReader._read_low_memory
    File "pandas\_libs\parsers.pyx", line 948, in pandas._libs.parsers.TextReader._read_rows
    File "pandas\_libs\parsers.pyx", line 935, in pandas._libs.parsers.TextReader._tokenize_rows
    File "pandas\_libs\parsers.pyx", line 2130, in pandas._libs.parsers.raise_parser_error
  pandas.errors.ParserError: Error tokenizing data. C error: Unknown error in IO callback
  </code></pre>
  
  <p><strong>What are possible causes of this error? Could it be memory-related? How can I troubleshoot this? Should I chunk this data?</strong></p>
  
  <p>I don't suspect a RAM issue, since I have over 7 GBs of unused RAM while the function is being called, as seen in my Windows 10 Task Manager Performance monitor. Also, I can't provide any examples of the underlying data, because it's health and PII data.</p>
  
  <h3>Here is the excerpt from my code:</h3>
  
  <pre><code>import os
  import pandas as pd
  
  # File
  filepath = "C:\\Temp\\datafile.txt"
  filename_w_ext = "datafile.txt"
  
  # Read in TXT file
  def read_in_files(filepath, filename_w_ext):
      filename, file_ext = os.path.splitext(filename_w_ext)
      print('Reading in file {}'.format(filepath))
      with open(filepath, "r", newline='') as file:
          global df_data
          # Here's where it errors:
          df_data = pd.read_csv(file, dtype=str, sep='|')
          return df_data.columns.values.tolist(), df_data.values.tolist()
  </code></pre>
  
  <p>Googling for this specific error only gives the source code for the error handling in the pandas <a href="https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/src/parser/tokenizer.c" rel="nofollow noreferrer">Tokenizer code</a></p>
  
  <h3>Lines 583-612:</h3>
  
  <pre><code>static int parser_buffer_bytes(parser_t *self, size_t nbytes) {
      int status;
      size_t bytes_read;
  
      status = 0;
      self-&gt;datapos = 0;
      self-&gt;data = self-&gt;cb_io(self-&gt;source, nbytes, &amp;bytes_read, &amp;status);
      TRACE((
          "parser_buffer_bytes self-&gt;cb_io: nbytes=%zu, datalen: %d, status=%d\n",
          nbytes, bytes_read, status));
      self-&gt;datalen = bytes_read;
  
      if (status != REACHED_EOF &amp;&amp; self-&gt;data == NULL) {
          int64_t bufsize = 200;
          self-&gt;error_msg = (char *)malloc(bufsize);
  
          if (status == CALLING_READ_FAILED) {
              snprintf(self-&gt;error_msg, bufsize,
                       "Calling read(nbytes) on source failed. "
                       "Try engine='python'.");
          } else {
              snprintf(self-&gt;error_msg, bufsize, "Unknown error in IO callback");
          }
          return -1;
      }
  
      TRACE(("datalen: %d\n", self-&gt;datalen));
  
      return status;
  }
  </code></pre>