<h4 dir="auto">Code Sample</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="df = pd.DataFrame([{'i': i, 's': str(i)} for i in range(5)])

# this gives two rows with counts of one, as expected
df.iloc[:2].groupby('s').size()

df['s'] = df['s'].astype('category')
# this gives five rows, two of those having counts of one and others of zero
df.iloc[:2].groupby('s').size()"><pre class="notranslate"><span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>([{<span class="pl-s">'i'</span>: <span class="pl-s1">i</span>, <span class="pl-s">'s'</span>: <span class="pl-en">str</span>(<span class="pl-s1">i</span>)} <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">5</span>)])

<span class="pl-c"># this gives two rows with counts of one, as expected</span>
<span class="pl-s1">df</span>.<span class="pl-s1">iloc</span>[:<span class="pl-c1">2</span>].<span class="pl-en">groupby</span>(<span class="pl-s">'s'</span>).<span class="pl-en">size</span>()

<span class="pl-s1">df</span>[<span class="pl-s">'s'</span>] <span class="pl-c1">=</span> <span class="pl-s1">df</span>[<span class="pl-s">'s'</span>].<span class="pl-en">astype</span>(<span class="pl-s">'category'</span>)
<span class="pl-c"># this gives five rows, two of those having counts of one and others of zero</span>
<span class="pl-s1">df</span>.<span class="pl-s1">iloc</span>[:<span class="pl-c1">2</span>].<span class="pl-en">groupby</span>(<span class="pl-s">'s'</span>).<span class="pl-en">size</span>()</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">I'm using categorical columns to save memory and possibly improve performance when having lots of strings, and would expect grouping by them to behave the same whether the string column is left as-is or converted to category. If that's expected behavior for categorical groupby to keep empty groups, is it possible to at least provide a boolean parameter to <code class="notranslate">groupby</code> like <code class="notranslate">empty_groups</code>? Or maybe even a simpler solution exists, but I couldn't find it.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.6.1.final.0<br>
python-bits: 64<br>
OS: Linux<br>
OS-release: 4.9.16-gentoo<br>
machine: x86_64<br>
processor: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: en_US.utf8<br>
LOCALE: en_US.UTF-8</p>
<p dir="auto">pandas: 0.20.2<br>
pytest: 3.1.2<br>
pip: 9.0.1<br>
setuptools: 27.2.0<br>
Cython: 0.25.2<br>
numpy: 1.12.1<br>
scipy: 0.19.1<br>
xarray: None<br>
IPython: 6.1.0<br>
sphinx: 1.6.2<br>
patsy: 0.4.1<br>
dateutil: 2.6.0<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: 1.2.1<br>
tables: 3.3.0<br>
numexpr: 2.6.2<br>
feather: None<br>
matplotlib: 2.0.2<br>
openpyxl: 2.4.7<br>
xlrd: 1.0.0<br>
xlwt: 1.2.0<br>
xlsxwriter: 0.9.6<br>
lxml: 3.8.0<br>
bs4: 4.6.0<br>
html5lib: 0.999<br>
sqlalchemy: 1.1.11<br>
pymysql: None<br>
psycopg2: 2.7.1 (dt dec pq3 ext lo64)<br>
jinja2: 2.9.6<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>
<h4 dir="auto">Code Sample</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pandas.DataFrame.duplicated(subset=None, keep='first')"><pre class="notranslate"><span class="pl-s1">pandas</span>.<span class="pl-v">DataFrame</span>.<span class="pl-en">duplicated</span>(<span class="pl-s1">subset</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">keep</span><span class="pl-c1">=</span><span class="pl-s">'first'</span>)</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">Pandas have a built-in method for denoting duplicated records. However, it is not able to denote duplicated record by some combination of keys.<br>
<strong>dataFrame a</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="index1  index2  val1  val2
a      y      1      1
a      y      1      2
b      z      2      2
c      x      3      3"><pre class="notranslate"><code class="notranslate">index1  index2  val1  val2
a      y      1      1
a      y      1      2
b      z      2      2
c      x      3      3
</code></pre></div>
<p dir="auto"><strong>dataFrame b</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="index1  index2  val3
a      y      1
b      z      2"><pre class="notranslate"><code class="notranslate">index1  index2  val3
a      y      1
b      z      2
</code></pre></div>
<p dir="auto">there are nodupkey for the first glance.<br>
but this could cause issues when performing merging with validate='1:1' criteria.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pd.merge(a, b, validate='1:1')"><pre class="notranslate"><code class="notranslate">pd.merge(a, b, validate='1:1')
</code></pre></div>
<p dir="auto">we expect that both data set is unique by index1 and index2.<br>
But in dataFrame a, the records are not unique which cause merging failed.<br>
In SAS, duplicate in by group can be found by executing</p>
<div class="highlight highlight-source-sas notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="proc sort data=sample nouniqueky out=dup; 
    by var1 var2; 
run;"><pre class="notranslate"><span class="pl-c1">proc <span class="pl-c1">sort</span></span> data=sample nouniqueky <span class="pl-c1"><span class="pl-c1">out</span></span>=dup; 
    <span class="pl-k">by</span> var1 var2; 
<span class="pl-k">run</span>;</pre></div>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">Expected add a parameter such as <code class="notranslate">groupby or index</code></p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.6.3.final.0<br>
python-bits: 64<br>
OS: Windows<br>
OS-release: 7<br>
machine: AMD64<br>
processor: Intel64 Family 6 Model 69 Stepping 1, GenuineIntel<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: None<br>
LOCALE: None.None</p>
<p dir="auto">pandas: 0.21.1<br>
pytest: 3.2.1<br>
pip: 9.0.1<br>
setuptools: 36.5.0.post20170921<br>
Cython: 0.26.1<br>
numpy: 1.13.3<br>
scipy: 0.19.1<br>
pyarrow: None<br>
xarray: None<br>
IPython: 6.1.0<br>
sphinx: 1.6.3<br>
patsy: 0.4.1<br>
dateutil: 2.6.1<br>
pytz: 2017.3<br>
blosc: None<br>
bottleneck: 1.2.1<br>
tables: 3.4.2<br>
numexpr: 2.6.2<br>
feather: None<br>
matplotlib: 2.1.0<br>
openpyxl: 2.4.8<br>
xlrd: 1.1.0<br>
xlwt: 1.3.0<br>
xlsxwriter: 1.0.2<br>
lxml: 4.1.0<br>
bs4: 4.6.0<br>
html5lib: 0.999999999<br>
sqlalchemy: 1.1.13<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.9.6<br>
s3fs: None<br>
fastparquet: None<br>
pandas_gbq: None<br>
pandas_datareader: 0.5.0</p>
</details>