Error tokenizing data
Question: This is my code:

    
    
    import pandas
    import datetime
    from decimal import Decimal
    
    file_ = open('myfile.csv', 'r')
    result = pandas.read_csv(
        file_, header=None,
        names=('sec', 'date', 'sale', 'buy'),
        usecols=('date', 'sale', 'buy'),
        parse_dates=['date'],
        iterator=True,
        chunksize=100,
        compression=None,
        engine="c",
        date_parser=lambda dt: datetime.datetime.strptime(dt, '%Y%m%d %H:%M:%S.%f'),
        converters={'sale': (lambda u: Decimal(u)), 'buy': (lambda u: Decimal(u))}
    )
    

And then I try...

    
    
    result.get_chunk()
    

Only to get an error like this:

    
    
    CParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 4
    

From a file like this (I just show the first 4 lines - the file has no header,
and all the lines have this format):

    
    
    EUR/USD,20160701 00:00:00.071,1.11031,1.11033
    EUR/USD,20160701 00:00:00.255,1.11031,1.11033
    EUR/USD,20160701 00:00:00.256,1.11025,1.11033
    EUR/USD,20160701 00:00:00.258,1.11027,1.11033
    ... > l0.000.000 lines like these
    

My intention is to get an object to iterate by chunks and not have the whole
crap in memory (the actual file has 560mb!). I want to discard the first
column (there are 4 columns but since this file has the same value in the
first column, I want to discard such column). I want to keep columns 1, 2, and
3 (discarding 0) as date, sale, and purchase price.

Actually this is my first attempt with pandas, since the former solution used
standard Python csv module, and takes a lot of time.

What am I missing? Why am I getting such error?


Answer: 
    #try this code
    import pandas as pd
    import numpy as np
    import csv
    
    # To print only three columns , create a data frame,to do that give names to columns in csv file with ',' as seperator
    myfile.csv:
    sec,date,sale,buy
    EUR/USD,20160701 00:00:00.071,1.11031,1.11033
    EUR/USD,20160701 00:00:00.255,1.11031,1.11033
    EUR/USD,20160701 00:00:00.256,1.11025,1.11033
    EUR/USD,20160701 00:00:00.258,1.11027,1.11033
    
    data = pd.read_csv('myfile.csv',sep=',')
    df = pd.DataFrame({'date':data.date,'sale':data.sale,'buy':data.buy})
    print(df)
    
    output:
           buy                   date     sale
    0  1.11033  20160701 00:00:00.071  1.11031
    1  1.11033  20160701 00:00:00.255  1.11031
    2  1.11033  20160701 00:00:00.256  1.11025
    3  1.11033  20160701 00:00:00.258  1.11027
    

