Ipython kernel dies unexpectedly reading large file
Question: I'm reading in a ~3Gb csv using pandas in an ipython notebook. While reading
the file, the notebook unexpectedly gives me an error message saying the
kernel appears to have died and will restart.

As per several "big data" workflows in python/pandas, I'm reading the files in
as follows:

    
    
    import pandas as pd
    tp = pd.read_csv(file_name_cleaned,chunksize,iterator=True,low_memory=False)
    df = pd.concat(tp,ignore_index=True)
    

My workflow has involved some preprocessing to remove all but alphanumeric
characters and a few pieces of punctuation as follows:

    
    
    with open(file_name,'r') as file1:
        with open(file_name_cleaned,'w') as file:2
            for line in file1:
                    if len(line.split(sep_string)) == num_columns:
                        line = re.sub(r'[^A-Za-z0-9|._]+','',line)
                        file2.write(line+'\n')
    

The strange thing is that if I remove the line containing re.sub(), I get a
different error - "Expected 209 fileds, saw in line 22236, saw 329" even
though I've explicitly checked for the exact number of delimiters. Visual
inspection of the line and surrounding lines don't really show me much either.

This process has worked fine for several other files, including ones that are
larger so I don't think the size of the file is the issue although I suppose
it's possible that that's an oversimplification.

I included the preprocessing because I know from experience that sometimes the
data contains strange special characters, I've also gone back and forth
between using encoding='utf-8' and encoding='utf-8-sig' in the read_csv() and
open() statements to no real avail.

I have several questions - does including the encoding keyword argument cause
python to ignore characters outside of those character sets or does it maybe
invoke some kind of conversion for those characters? I'm not very familiar
with these types of issues. Is it possible that some kind of unexpected
character could have slipped through my preprocessing and caused this? Is
there another type of issue that I haven't found that could cause this? (I
have done research but nothing has been quite right.)

Any help would be much appreciated.

Also, I'm using Anaconda 2.4, with Python 3.5.1, Ipython 4.0.0, and pandas
0.17.0


Answer: I'm not sure that this totally answers my questions but I did solve the issue,
while it is slower, using engine='python' in pd.read_csv() did the trick.

