<p dir="auto">getting pretty long, could create a sub-dir and split out into separate files.</p>
<p dir="auto">Dataframe.duplicated() is flagging rows as duplicates when they are in fact distinct. This happens when using large dataframes, and duplicated(keep=False):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import pandas as pd, numpy as np
df = pd.DataFrame({'a': pd.Series(range(1,100000)),
                   'b': pd.Series(range(10,1000000)),
                   'c': pd.Series(3*range(2,200000,2))})
df.head()

np.sum(df.duplicated())"><pre class="notranslate"><code class="notranslate">import pandas as pd, numpy as np
df = pd.DataFrame({'a': pd.Series(range(1,100000)),
                   'b': pd.Series(range(10,1000000)),
                   'c': pd.Series(3*range(2,200000,2))})
df.head()

np.sum(df.duplicated())
</code></pre></div>
<p dir="auto">Out[]: 0</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="np.sum(df.duplicated(keep=False))"><pre class="notranslate"><code class="notranslate">np.sum(df.duplicated(keep=False))
</code></pre></div>
<p dir="auto">Out[]:110</p>
<p dir="auto">Changing column order results in different (but still incorrect) behavior.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="np.sum(df[['c','b','a']].duplicated(keep=False))"><pre class="notranslate"><code class="notranslate">np.sum(df[['c','b','a']].duplicated(keep=False))
</code></pre></div>
<p dir="auto">Out[]:2138</p>
<p dir="auto">Tested on 0.17.1. Environment details are provided below:<br>
&gt;&gt; pd.util.print_versions.show_versions()<br>
INSTALLED VERSIONS<br>
------------------<br>
commit: None<br>
python: 2.7.11.final.0<br>
python-bits: 64<br>
OS: Darwin<br>
OS-release: 14.5.0<br>
machine: x86_64<br>
processor: i386<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: en_GB.UTF-8</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pandas: 0.17.1
nose: None
pip: 7.1.2
setuptools: 18.5
Cython: None
numpy: 1.10.1
scipy: 0.16.0
statsmodels: None
IPython: 4.0.0
sphinx: 1.3.3
patsy: None
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.9
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext)
Jinja2: None"><pre class="notranslate"><code class="notranslate">pandas: 0.17.1
nose: None
pip: 7.1.2
setuptools: 18.5
Cython: None
numpy: 1.10.1
scipy: 0.16.0
statsmodels: None
IPython: 4.0.0
sphinx: 1.3.3
patsy: None
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.9
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext)
Jinja2: None
</code></pre></div>
<p dir="auto">This looks like the same kind of problem described in  <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="118095000" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/11668" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/11668/hovercard" href="https://github.com/pandas-dev/pandas/issues/11668">#11668</a>, though the specific examples provided in that issue work properly in 0.17.1</p>