Title: Read all but last line of CSV file in pandas
Tags: python;pandas;dataframe
Question: I have CSV files which I read in in pandas with:
#!/usr/bin/env python
import pandas as pd
import sys
filename = sys.argv[1]
df = pd.read_csv(filename)
Unfortunately, the last line of these files is often corrupt (has the wrong number of commas).  Currently I open each file in a text editor and remove the last line.
Is it possible to remove the last line in the same python/pandas script that loads the CSV to save having to take this extra non-automated step?
Comment: You deleted a question about extracting numbers, anyway I was going to suggest using `str.extract`: `for col in df.columns[2:]:
df[col] = df[col].str.extract(r'(\d+)').astype(int)`

Comment: @EdChum Does your code leave the decimal points?

Comment: @EdChum I undeleted the previous question.

Here is the accepted answer: pass error_bad_lines=False and it will skip this line automatically
df = pd.read_csv(filename, error_bad_lines=False)
The advantage of error_bad_lines is it will skip and not bork on any erroneous lines but if the last line is always duff then skipfooter=1 is better
Thanks to @DexterMorgan for pointing out that skipfooter option forces the engine to use the python engine which is slower than the c engine for parsing a csv.
Comment for this answer: There's an option 'skiprows', which is supported with c engine. If you know the number of lines of your csv you could add it as follows:  `pd.read_csv(filename, skiprows=[999])` (In my case there are 1000 lines) - note that you have to define rows in a list if you want to specify rows given their line number.
Comment for this answer: @EdChum I'm just looking into the documentation because I need it right now and I didn't see this option until now. Thank you!
So In my case instead of `skiprows=[1000]` I had to write `nrows=999`.
Another solution could be removing the last line via the command line which is very fast: `head -n -1 dataframe.csv > temp.csv && mv temp.csv dataframe.csv`
Comment for this answer: @DexterMorgan sure will add
Comment for this answer: @Chaoste but the bad rows are at the end though, wouldn't you want `nrows` instead?
Comment for this answer: Sure, the docs are pretty good, I think with `error_bad_lines=False` it should work fine with the c engine and be fast
Comment for this answer: Regarding the `skipfooter` option, it might be good to know that it doesn't work with the `dtypes` option:
`ValueError: Falling back to the 'python' engine because the 'c' engine does not support skipfooter, but this causes 'dtype' to be ignored as it is not supported by the 'python' engine. (Note the 'converters' option provides similar functionality.)`

Here is another answer: You can leave out the last n lines when reading in a csv by using the skipfooter argument:
df = pd.read_csv(filename, skipfooter=3, engine='python')
In this example the last 3 lines are ommited.

Here is another answer: Read http://pandas.pydata.org/pandas-docs/version/0.16.2/generated/pandas.read_csv.html. Here 'skipfooter' argument can be used to specify no of lines that you don't want to read from .csv file from the end. May be It may help you.

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 3, 'CommentCount': 3, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2015:00:11/13/15', 'FavoriteCount': 2.0, 'PostTypeId': 1, 'Tags': '<python><pandas><dataframe>', 'Title': 'Read all but last line of CSV file in pandas', 'ViewCount': 37367, 'id': 33689694, 'user_AboutMe': None, 'user_AccountId': 514072.0, 'user_DisplayName': 'graffe', 'user_Reputation': 17716.0, 'user_Views': 1372.0, 'user_id': 1473517.0}}