Error tokenizing data. C error: EOF following escape character
Question: I'm trying to load a csv text file that I created with an OS X app written in
Objective-C (using XCode). The text file (temp2.csv) looks fine in an editor
but there's something wrong with it and I get this error when reading it into
a Pandas dataframe. If I copy the data into a fresh text file (temp.csv) and
save that it works fine! The two text files are clearly different (one is 74
bytes the other is 150) - invisible characters perhaps? - but it's very
annoying as I want the python code to load the text files produced by the C
code. Files are attached for reference.

temp.csv

    
    
    -3.132700,0.355885,9.000000,0.444416
    -3.128256,0.444416,9.000000,0.532507
    

temp2.csv

    
    
    -3.132700,0.355885,9.000000,0.444416
    -3.128256,0.444416,9.000000,0.532507
    

(I can't find any help on this specific error on StackExchange).

    
    
    Python 2.7.11 |Anaconda 2.2.0 (x86_64)| (default, Dec  6 2015, 18:57:58) 
    [GCC 4.2.1 (Apple Inc. build 5577)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    Anaconda is brought to you by Continuum Analytics.
    Please check out: http://continuum.io/thanks and https://anaconda.org
    >>> import pandas as pd
    >>> df = pd.read_csv("temp2.csv", header=None)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/billtubbs/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 498, in parser_f
        return _read(filepath_or_buffer, kwds)
      File "/Users/billtubbs/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 275, in _read
        parser = TextFileReader(filepath_or_buffer, **kwds)
      File "/Users/billtubbs/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 590, in __init__
        self._make_engine(self.engine)
      File "/Users/billtubbs/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 731, in _make_engine
        self._engine = CParserWrapper(self.f, **self.options)
      File "/Users/billtubbs/anaconda/lib/python2.7/site-packages/pandas/io/parsers.py", line 1103, in __init__
        self._reader = _parser.TextReader(src, **kwds)
      File "pandas/parser.pyx", line 515, in pandas.parser.TextReader.__cinit__ (pandas/parser.c:4948)
      File "pandas/parser.pyx", line 717, in pandas.parser.TextReader._get_header (pandas/parser.c:7496)
      File "pandas/parser.pyx", line 829, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8838)
      File "pandas/parser.pyx", line 1833, in pandas.parser.raise_parser_error (pandas/parser.c:22649)
    pandas.parser.CParserError: Error tokenizing data. C error: EOF following escape character
    >>> df = pd.read_csv("temp.csv", header=None)
    >>> df
              0         1  2         3
    0 -3.132700  0.355885  9  0.444416
    1 -3.128256  0.444416  9  0.532507
    

Footnote: I think I located the problem.

    
    
    >>> f = open('temp2.csv')
    >>> contents = f.read()
    >>> print contents
    ??-3.132700,0.355885,9.000000,0.444416
    -3.128256,0.444416,9.000000,0.532507
    >>> contents
    '\xff\xfe-\x003\x00.\x001\x003\x002\x007\x000\x000\x00,\x000\x00.\x003\x005\x005\x008\x008\x005\x00,\x009\x00.\x000\x000\x000\x000\x000\x000\x00,\x000\x00.\x004\x004\x004\x004\x001\x006\x00\n\x00-\x003\x00.\x001\x002\x008\x002\x005\x006\x00,\x000\x00.\x004\x004\x004\x004\x001\x006\x00,\x009\x00.\x000\x000\x000\x000\x000\x000\x00,\x000\x00.\x005\x003\x002\x005\x000\x007\x00'
    

It's full of escape characters! How to remove them?


Answer: You need add parameter `encoding` to
[`read_csv`](http://pandas.pydata.org/pandas-
docs/stable/generated/pandas.read_csv.html), because file encoding is
`UTF-16`:

    
    
    import pandas as pd
    
    contents = '\xff\xfe-\x003\x00.\x001\x003\x002\x007\x000\x000\x00,\x000\x00.\x003\x005\x005\x008\x008\x005\x00,\x009\x00.\x000\x000\x000\x000\x000\x000\x00,\x000\x00.\x004\x004\x004\x004\x001\x006\x00\n\x00-\x003\x00.\x001\x002\x008\x002\x005\x006\x00,\x000\x00.\x004\x004\x004\x004\x001\x006\x00,\x009\x00.\x000\x000\x000\x000\x000\x000\x00,\x000\x00.\x005\x003\x002\x005\x000\x007\x00'
    
    text_file = open("test/file1.csv", "wb")
    text_file.write(contents)
    text_file.close()
    
    df = pd.read_csv("test/file1.csv", header=None, encoding='utf-16')
    print df
    
              0         1  2         3
    0 -3.132700  0.355885  9  0.444416
    1 -3.128256  0.444416  9  0.532507
    

