<p dir="auto">CSV files can contains some errors, for example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME,PAT
Peter,cat
really bad line
Fedor,cat"><pre class="notranslate"><code class="notranslate">NAME,PAT
Peter,cat
really bad line
Fedor,cat
</code></pre></div>
<p dir="auto">to skip really bad lines exist <code class="notranslate">error_bad_lines=False</code> parameter.<br>
Another example without quotes and delimiters in field:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME,PAT
Peter,cat
Ira,cat,dog
Fedor,cat"><pre class="notranslate"><code class="notranslate">NAME,PAT
Peter,cat
Ira,cat,dog
Fedor,cat
</code></pre></div>
<p dir="auto">Which with quotes will look like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME,PAT
Peter,cat
Ira,&quot;cat,dog&quot;
Fedor,cat"><pre class="notranslate"><code class="notranslate">NAME,PAT
Peter,cat
Ira,"cat,dog"
Fedor,cat
</code></pre></div>
<p dir="auto">So it easy fix this line if know that first field not contain extra separators.<br>
Also extra trailing delimiters issue: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="11082057" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/2886" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/2886/hovercard" href="https://github.com/pandas-dev/pandas/issues/2886">#2886</a></p>
<p dir="auto">More real life example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="82,52,29,11,2,2013-08-02 00:00:00,,,gen,,FDP, employee,0,1,gen,,,0
55,69,36,19,2,2013-10-28 00:00:00,,,gen,,FDP employee,0,1,gen,,,0"><pre class="notranslate"><code class="notranslate">82,52,29,11,2,2013-08-02 00:00:00,,,gen,,FDP, employee,0,1,gen,,,0
55,69,36,19,2,2013-10-28 00:00:00,,,gen,,FDP employee,0,1,gen,,,0
</code></pre></div>
<p dir="auto">There are difference for <code class="notranslate">FDP employee</code> and <code class="notranslate">FDP, employee</code>. So it will be grate to have ability process this bad lines with own handler.</p>
<p dir="auto">My proposition add additional parameter <code class="notranslate">process_bad_lines</code> for <code class="notranslate">read_csv</code>.<br>
For example, if I want fix line:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def bad_line_handler(items):
    '''probably ugly example,
       but lets imagine that `FDP, employee` is half of our data'''
    fdp_index = items.index('FDP')
    return items[:fdp_index] + ['FDP, employee'] + items[fdp_index + 2:]

pd.read_csv(file, process_bad_lines=bad_line_handler)"><pre class="notranslate"><code class="notranslate">def bad_line_handler(items):
    '''probably ugly example,
       but lets imagine that `FDP, employee` is half of our data'''
    fdp_index = items.index('FDP')
    return items[:fdp_index] + ['FDP, employee'] + items[fdp_index + 2:]

pd.read_csv(file, process_bad_lines=bad_line_handler)
</code></pre></div>
<ol dir="auto">
<li><code class="notranslate">error_bad_line</code> and <code class="notranslate">warn_bad_line</code> can work as before but at first once try replace bad string with <code class="notranslate">process_bad_lines</code> handler.</li>
<li>if <code class="notranslate">process_bad_lines</code> will return <code class="notranslate">None</code> when probably better just skip this line without exceptions (probably it more flexible), to store compatibility just return unchanged <code class="notranslate">items</code> parameter. Otherwise <code class="notranslate">None</code> can be equal to bad line and better raise exceptions from <code class="notranslate">process_bad_lines</code> handler.</li>
<li>not always I can already create CSV file with quotes, somebody already send me bad CSV.</li>
<li>I can pre-process file but it will take more time and work, see for example <a href="http://stackoverflow.com/questions/14550441/problems-reading-csv-file-with-commas-and-characters-in-pandas" rel="nofollow">http://stackoverflow.com/questions/14550441/problems-reading-csv-file-with-commas-and-characters-in-pandas</a></li>
</ol>
<p dir="auto">Some additions:<br>
For example I have no much string fields and can assume that one of strings contains separator:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="int,int,int,int,int,datetime,,,str,,str,str,int,int,str,,,int
int,int,int,int,int,datetime,,,str,,str    ,int,int,str,,,int"><pre class="notranslate"><code class="notranslate">int,int,int,int,int,datetime,,,str,,str,str,int,int,str,,,int
int,int,int,int,int,datetime,,,str,,str    ,int,int,str,,,int
</code></pre></div>
<p dir="auto">But it can work bad for many strings:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="int,int,int,int,int,datetime,str,str,str,str,str,str,int,int,str,,,int
int,int,int,int,int,datetime,  str,str,str,str,str  ,int,int,str,,,int"><pre class="notranslate"><code class="notranslate">int,int,int,int,int,datetime,str,str,str,str,str,str,int,int,str,,,int
int,int,int,int,int,datetime,  str,str,str,str,str  ,int,int,str,,,int
</code></pre></div>
<p dir="auto">However it also be grate have default methods to fix this strings with concatenating left strings:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="int,int,int,int,int,datetime,str,str,str,str,str,str,int,int,str,,,int
int,int,int,int,int,datetime,  STR  ,str,str,str,str,int,int,str,,,int

# for example with next syntax
pd.read_csv(file, process_bad_lines='try_concat_left')"><pre class="notranslate"><code class="notranslate">int,int,int,int,int,datetime,str,str,str,str,str,str,int,int,str,,,int
int,int,int,int,int,datetime,  STR  ,str,str,str,str,int,int,str,,,int

# for example with next syntax
pd.read_csv(file, process_bad_lines='try_concat_left')
</code></pre></div>
<p dir="auto">or right strings:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="int,int,int,int,int,datetime,str,str,str,str,str,str,int,int,str,,,int
int,int,int,int,int,datetime,str,str,str,str,  STR  ,int,int,str,,,int

# for example with next syntax
pd.read_csv(file, process_bad_lines='try_concat_right')"><pre class="notranslate"><code class="notranslate">int,int,int,int,int,datetime,str,str,str,str,str,str,int,int,str,,,int
int,int,int,int,int,datetime,str,str,str,str,  STR  ,int,int,str,,,int

# for example with next syntax
pd.read_csv(file, process_bad_lines='try_concat_right')
</code></pre></div>
<p dir="auto">and also removing extra trailing delimiters:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pd.read_csv(file, process_bad_lines='skip_right_delimiters')"><pre class="notranslate"><code class="notranslate">pd.read_csv(file, process_bad_lines='skip_right_delimiters')
</code></pre></div>
<p dir="auto">Lets say we have data with both column and row headers. Lets say this dataset also simply outputs a <code class="notranslate">0</code> in that csv file for the cell at the intersection of the column and row headers (A1 in excel notation, or cell (0,0) in the csv file). Additionally, both <code class="notranslate">"0"</code> and <code class="notranslate">"0.1"</code> are valid column names:</p>
<table role="table">
<thead>
<tr>
<th>RH\CH</th>
<th>-0.1</th>
<th>0</th>
<th>0.1</th>
</tr>
</thead>
<tbody>
<tr>
<td>10.0</td>
<td>123</td>
<td>456</td>
<td>789</td>
</tr>
<tr>
<td>20.0</td>
<td>012</td>
<td>345</td>
<td>678</td>
</tr>
</tbody>
</table>
<p dir="auto">Thus, "RH\CH" is replaced by <code class="notranslate">"0"</code> on export (of which I have no control).</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
from cStringIO import StringIO

data = '0,-0.1,0,0.1\n10.0,123,456,789\n20.0,012,345,678'
df = pd.read_csv(StringIO(data),index_col=0,header=0)
print(df)

      -0.1  0.1  0.1
0                   
10.0   123  789  789
20.0    12  678  678
"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">from</span> <span class="pl-s1">cStringIO</span> <span class="pl-k">import</span> <span class="pl-v">StringIO</span>

<span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s">'0,-0.1,0,0.1<span class="pl-cce">\n</span>10.0,123,456,789<span class="pl-cce">\n</span>20.0,012,345,678'</span>
<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-v">StringIO</span>(<span class="pl-s1">data</span>),<span class="pl-s1">index_col</span><span class="pl-c1">=</span><span class="pl-c1">0</span>,<span class="pl-s1">header</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)
<span class="pl-en">print</span>(<span class="pl-s1">df</span>)

      <span class="pl-c1">-</span><span class="pl-c1">0.1</span>  <span class="pl-c1">0.1</span>  <span class="pl-c1">0.1</span>
<span class="pl-c1">0</span>                   
<span class="pl-c1">10.0</span>   <span class="pl-c1">123</span>  <span class="pl-c1">789</span>  <span class="pl-c1">789</span>
<span class="pl-c1">20.0</span>    <span class="pl-c1">12</span>  <span class="pl-c1">678</span>  <span class="pl-c1">678</span></pre></div>
<p dir="auto">The name mangling will change the duplicate <code class="notranslate">"0"</code> column containing data to <code class="notranslate">"0.1"</code>, thus if a real data column has the name <code class="notranslate">"0.1"</code>, this data will be copied (?) back to the mangled duplicate <code class="notranslate">"0"</code> column. Additionally, if the true <code class="notranslate">"0.1"</code> data name is missing, then this name mangling is frustrating as there is no obvious way to determine if the now present <code class="notranslate">"0.1"</code> column is a duplicate <code class="notranslate">"0"</code> that has been mangled or is a real <code class="notranslate">"0.1"</code> data series.</p>
<p dir="auto">I propose the addition of a <code class="notranslate">mangle_dupe_cols_str</code> keyword option that defaults to <code class="notranslate">'.'</code> to preserve the current behavior. However, it can be passed as a kwarg to <code class="notranslate">read_csv</code> in cases where the period name mangling could result in further duplicate columns.</p>
<p dir="auto">In <a href="https://github.com/pandas-dev/pandas/blob/v0.19.1/pandas/io/parsers.py#L2109-L2111">https://github.com/pandas-dev/pandas/blob/v0.19.1/pandas/io/parsers.py#L2109-L2111</a>:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="                        if cur_count &gt; 0:
                            this_columns[i] = '%s.%d' % (col, cur_count)
                        counts[col] = cur_count + 1"><pre class="notranslate">                        <span class="pl-k">if</span> <span class="pl-s1">cur_count</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">0</span>:
                            <span class="pl-s1">this_columns</span>[<span class="pl-s1">i</span>] <span class="pl-c1">=</span> <span class="pl-s">'%s.%d'</span> <span class="pl-c1">%</span> (<span class="pl-s1">col</span>, <span class="pl-s1">cur_count</span>)
                        <span class="pl-s1">counts</span>[<span class="pl-s1">col</span>] <span class="pl-c1">=</span> <span class="pl-s1">cur_count</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span></pre></div>
<p dir="auto">should be adapted to</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="                        if cur_count &gt; 0:
                            this_columns[i] = '%s%s%d' % (col, self.mangle_dupe_cols_str, cur_count)
                        counts[col] = cur_count + 1"><pre class="notranslate">                        <span class="pl-k">if</span> <span class="pl-s1">cur_count</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">0</span>:
                            <span class="pl-s1">this_columns</span>[<span class="pl-s1">i</span>] <span class="pl-c1">=</span> <span class="pl-s">'%s%s%d'</span> <span class="pl-c1">%</span> (<span class="pl-s1">col</span>, <span class="pl-s1">self</span>.<span class="pl-s1">mangle_dupe_cols_str</span>, <span class="pl-s1">cur_count</span>)
                        <span class="pl-s1">counts</span>[<span class="pl-s1">col</span>] <span class="pl-c1">=</span> <span class="pl-s1">cur_count</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span></pre></div>
<p dir="auto">and lines 1042-1043 in <a href="https://github.com/pandas-dev/pandas/blob/v0.19.1/pandas/io/parsers.py#L1042-L1043">https://github.com/pandas-dev/pandas/blob/v0.19.1/pandas/io/parsers.py#L1042-L1043</a> changed to:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="        self.mangle_dupe_cols = kwds.get('mangle_dupe_cols', True)
        self.mangle_dupe_cols_str = kwds.get('mangle_dupe_cols_str', '.')        
        self.infer_datetime_format = kwds.pop('infer_datetime_format', False)"><pre class="notranslate">        <span class="pl-s1">self</span>.<span class="pl-s1">mangle_dupe_cols</span> <span class="pl-c1">=</span> <span class="pl-s1">kwds</span>.<span class="pl-en">get</span>(<span class="pl-s">'mangle_dupe_cols'</span>, <span class="pl-c1">True</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">mangle_dupe_cols_str</span> <span class="pl-c1">=</span> <span class="pl-s1">kwds</span>.<span class="pl-en">get</span>(<span class="pl-s">'mangle_dupe_cols_str'</span>, <span class="pl-s">'.'</span>)        
        <span class="pl-s1">self</span>.<span class="pl-s1">infer_datetime_format</span> <span class="pl-c1">=</span> <span class="pl-s1">kwds</span>.<span class="pl-en">pop</span>(<span class="pl-s">'infer_datetime_format'</span>, <span class="pl-c1">False</span>)</pre></div>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">Then the corrected output would be:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; data = '0,-0.1,0,0.1\n10.0,123,456,789\n20.0,012,345,678'
&gt;&gt;&gt; df = pd.read_csv(StringIO(data),index_col=0,header=0,mangle_dupe_cols_str='__')
&gt;&gt;&gt; print(df)

      -0.1  0__1  0.1
0                   
10.0   123  456  789
20.0    12  345  678
"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s">'0,-0.1,0,0.1<span class="pl-cce">\n</span>10.0,123,456,789<span class="pl-cce">\n</span>20.0,012,345,678'</span>
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-v">StringIO</span>(<span class="pl-s1">data</span>),<span class="pl-s1">index_col</span><span class="pl-c1">=</span><span class="pl-c1">0</span>,<span class="pl-s1">header</span><span class="pl-c1">=</span><span class="pl-c1">0</span>,<span class="pl-s1">mangle_dupe_cols_str</span><span class="pl-c1">=</span><span class="pl-s">'__'</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">print</span>(<span class="pl-s1">df</span>)

      <span class="pl-c1">-</span><span class="pl-c1">0.1</span>  <span class="pl-c1">0_</span><span class="pl-s1">_1</span>  <span class="pl-c1">0.1</span>
<span class="pl-c1">0</span>                   
<span class="pl-c1">10.0</span>   <span class="pl-c1">123</span>  <span class="pl-c1">456</span>  <span class="pl-c1">789</span>
<span class="pl-c1">20.0</span>    <span class="pl-c1">12</span>  <span class="pl-c1">345</span>  <span class="pl-c1">678</span></pre></div>
<p dir="auto">where subsequent operations could identify any columns with the mangle_dupe_cols_str, if needed. The current behavior silently causes data loss for certain column names.</p>
<details>
&gt;&gt;&gt; pd.show_versions()
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 2.7.12.final.0<br>
python-bits: 64<br>
OS: Windows<br>
OS-release: 7<br>
machine: AMD64<br>
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: en_US</p>
<p dir="auto">pandas: 0.18.1<br>
nose: 1.3.7<br>
pip: 8.1.2<br>
setuptools: None<br>
Cython: 0.24.1<br>
numpy: 1.11.1<br>
scipy: 0.18.1<br>
statsmodels: None<br>
xarray: None<br>
IPython: 5.1.0<br>
sphinx: 1.4.6<br>
patsy: 0.4.1<br>
dateutil: 2.5.3<br>
pytz: 2016.6.1<br>
blosc: None<br>
bottleneck: 1.1.0<br>
tables: 3.2.2<br>
numexpr: 2.6.1<br>
matplotlib: 1.5.3<br>
openpyxl: 2.3.2<br>
xlrd: 1.0.0<br>
xlwt: 1.1.2<br>
xlsxwriter: 0.9.3<br>
lxml: 3.6.4<br>
bs4: 4.5.1<br>
html5lib: None<br>
httplib2: None<br>
apiclient: None<br>
sqlalchemy: 1.1.3<br>
pymysql: None<br>
psycopg2: 2.6.2 (dt dec pq3 ext lo64)<br>
jinja2: 2.8<br>
boto: 2.42.0<br>
pandas_datareader: None</p>
</details>