<p dir="auto">At least for large datasets <code class="notranslate">DataFrame.duplicated</code> returns incorrect results.</p>
<p dir="auto">Consider MovieLens10M data (this code will automatically download the data from grouplens website):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
from requests import get
from StringIO import StringIO
zip_file_url = 'http://files.grouplens.org/datasets/movielens/ml-10m.zip'
zip_response = get(zip_file_url)
zip_contents = StringIO(zip_response.content)

with ZipFile(zip_contents) as zfile:
    zdata = zfile.read('ml-10M100K/ratings.dat')
    delimiter = ';'
    zdata = zdata.replace('::', delimiter) # makes data compatible with pandas c-engine
    mldata = pd.read_csv(StringIO(zdata), sep=delimiter, header=None, engine='c',
                              names=['userid', 'movieid', 'rating', 'timestamp'],
                              usecols=['userid', 'movieid', 'rating'])"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">from</span> <span class="pl-s1">requests</span> <span class="pl-k">import</span> <span class="pl-s1">get</span>
<span class="pl-k">from</span> <span class="pl-v">StringIO</span> <span class="pl-k">import</span> <span class="pl-v">StringIO</span>
<span class="pl-s1">zip_file_url</span> <span class="pl-c1">=</span> <span class="pl-s">'http://files.grouplens.org/datasets/movielens/ml-10m.zip'</span>
<span class="pl-s1">zip_response</span> <span class="pl-c1">=</span> <span class="pl-en">get</span>(<span class="pl-s1">zip_file_url</span>)
<span class="pl-s1">zip_contents</span> <span class="pl-c1">=</span> <span class="pl-v">StringIO</span>(<span class="pl-s1">zip_response</span>.<span class="pl-s1">content</span>)

<span class="pl-k">with</span> <span class="pl-v">ZipFile</span>(<span class="pl-s1">zip_contents</span>) <span class="pl-k">as</span> <span class="pl-s1">zfile</span>:
    <span class="pl-s1">zdata</span> <span class="pl-c1">=</span> <span class="pl-s1">zfile</span>.<span class="pl-en">read</span>(<span class="pl-s">'ml-10M100K/ratings.dat'</span>)
    <span class="pl-s1">delimiter</span> <span class="pl-c1">=</span> <span class="pl-s">';'</span>
    <span class="pl-s1">zdata</span> <span class="pl-c1">=</span> <span class="pl-s1">zdata</span>.<span class="pl-en">replace</span>(<span class="pl-s">'::'</span>, <span class="pl-s1">delimiter</span>) <span class="pl-c"># makes data compatible with pandas c-engine</span>
    <span class="pl-s1">mldata</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">read_csv</span>(<span class="pl-v">StringIO</span>(<span class="pl-s1">zdata</span>), <span class="pl-s1">sep</span><span class="pl-c1">=</span><span class="pl-s1">delimiter</span>, <span class="pl-s1">header</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">engine</span><span class="pl-c1">=</span><span class="pl-s">'c'</span>,
                              <span class="pl-s1">names</span><span class="pl-c1">=</span>[<span class="pl-s">'userid'</span>, <span class="pl-s">'movieid'</span>, <span class="pl-s">'rating'</span>, <span class="pl-s">'timestamp'</span>],
                              <span class="pl-s1">usecols</span><span class="pl-c1">=</span>[<span class="pl-s">'userid'</span>, <span class="pl-s">'movieid'</span>, <span class="pl-s">'rating'</span>])</pre></div>
<p dir="auto">The data (<code class="notranslate">mldata</code> variable) contains no duplicates, which can be verified:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="(mldata.groupby(['userid', 'movieid']).size()&gt;1).any()
False

mldata.set_index(['userid', 'movieid']).index.is_unique
True"><pre class="notranslate">(<span class="pl-s1">mldata</span>.<span class="pl-en">groupby</span>([<span class="pl-s">'userid'</span>, <span class="pl-s">'movieid'</span>]).<span class="pl-en">size</span>()<span class="pl-c1">&gt;</span><span class="pl-c1">1</span>).<span class="pl-en">any</span>()
<span class="pl-c1">False</span>

<span class="pl-s1">mldata</span>.<span class="pl-en">set_index</span>([<span class="pl-s">'userid'</span>, <span class="pl-s">'movieid'</span>]).<span class="pl-s1">index</span>.<span class="pl-s1">is_unique</span>
<span class="pl-c1">True</span></pre></div>
<p dir="auto">However,  <code class="notranslate">DataFrame.duplicated</code> gives:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="dups = mldata.duplicated(['userid', 'movieid'], keep=False)
print dups.any()
print dups.sum()

True
12127"><pre class="notranslate"><span class="pl-s1">dups</span> <span class="pl-c1">=</span> <span class="pl-s1">mldata</span>.<span class="pl-en">duplicated</span>([<span class="pl-s">'userid'</span>, <span class="pl-s">'movieid'</span>], <span class="pl-s1">keep</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
<span class="pl-k">print</span> <span class="pl-s1">dups</span>.<span class="pl-en">any</span>()
<span class="pl-k">print</span> <span class="pl-s1">dups</span>.<span class="pl-en">sum</span>()

<span class="pl-c1">True</span>
<span class="pl-c1">12127</span></pre></div>
<p dir="auto">Expected:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="False
0"><pre class="notranslate"><span class="pl-c1">False</span>
<span class="pl-c1">0</span></pre></div>
<p dir="auto"><code class="notranslate">pd.show_versions()</code>:</p>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 2.7.10.final.0<br>
python-bits: 64<br>
OS: Windows<br>
OS-release: 8<br>
machine: AMD64<br>
processor: Intel64 Family 6 Model 60 Stepping 3, GenuineIntel<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: None</p>
<p dir="auto">pandas: 0.17.0<br>
nose: 1.3.7<br>
pip: 7.1.2<br>
setuptools: 18.3.2<br>
Cython: 0.23.3<br>
numpy: 1.10.0<br>
scipy: 0.16.0<br>
statsmodels: None<br>
IPython: 3.2.1<br>
sphinx: 1.3.1<br>
patsy: 0.3.0<br>
dateutil: 2.4.2<br>
pytz: 2015.6<br>
blosc: None<br>
bottleneck: 1.0.0<br>
tables: 3.2.0<br>
numexpr: 2.3.1<br>
matplotlib: 1.4.3<br>
openpyxl: 1.8.5<br>
xlrd: 0.9.3<br>
xlwt: 1.0.0<br>
xlsxwriter: 0.7.3<br>
lxml: 3.4.4<br>
bs4: 4.3.2<br>
html5lib: 0.999<br>
httplib2: None<br>
apiclient: None<br>
sqlalchemy: 1.0.5<br>
pymysql: None<br>
psycopg2: None</p>
<h4 dir="auto">Code Sample</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd

df = pd.DataFrame([1,2,3], columns=['column'], index=[1,1,2])

for idx in df.index.unique():    
    print(type(df.loc[idx]['column']))
    for v in df.loc[idx]['column']:
        print(v)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>

<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>([<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">3</span>], <span class="pl-s1">columns</span><span class="pl-c1">=</span>[<span class="pl-s">'column'</span>], <span class="pl-s1">index</span><span class="pl-c1">=</span>[<span class="pl-c1">1</span>,<span class="pl-c1">1</span>,<span class="pl-c1">2</span>])

<span class="pl-k">for</span> <span class="pl-s1">idx</span> <span class="pl-c1">in</span> <span class="pl-s1">df</span>.<span class="pl-s1">index</span>.<span class="pl-en">unique</span>():    
    <span class="pl-en">print</span>(<span class="pl-en">type</span>(<span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[<span class="pl-s1">idx</span>][<span class="pl-s">'column'</span>]))
    <span class="pl-k">for</span> <span class="pl-s1">v</span> <span class="pl-c1">in</span> <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[<span class="pl-s1">idx</span>][<span class="pl-s">'column'</span>]:
        <span class="pl-en">print</span>(<span class="pl-s1">v</span>)</pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;class 'pandas.core.series.Series'&gt;
1
2
&lt;class 'numpy.int64'&gt;
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-26-a207307b6a42&gt; in &lt;module&gt;()
      5 for idx in df.index.unique():
      6     print(type(df.loc[idx]['column']))
----&gt; 7     for v in df.loc[idx]['column']:
      8         print(v)

TypeError: 'numpy.int64' object is not iterable"><pre class="notranslate"><code class="notranslate">&lt;class 'pandas.core.series.Series'&gt;
1
2
&lt;class 'numpy.int64'&gt;
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-26-a207307b6a42&gt; in &lt;module&gt;()
      5 for idx in df.index.unique():
      6     print(type(df.loc[idx]['column']))
----&gt; 7     for v in df.loc[idx]['column']:
      8         print(v)

TypeError: 'numpy.int64' object is not iterable
</code></pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">The problem with this code is, that <code class="notranslate">data.loc[idx]['column']</code> can either be a <code class="notranslate">Series</code> or a <code class="notranslate">float64</code> instance. This is a problem because this requires an additional check. Also this can lead to bugs very easily.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">I would expect this to return <em>always</em> a <code class="notranslate">Series</code> object.</p>