<p dir="auto">xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="46855603" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/8640" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/8640/hovercard" href="https://github.com/pandas-dev/pandas/issues/8640">#8640</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="142872513" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/12699" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/12699/hovercard" href="https://github.com/pandas-dev/pandas/issues/12699">#12699</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="158501831" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/13361" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/13361/hovercard" href="https://github.com/pandas-dev/pandas/pull/13361">#13361</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="159399596" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/13410" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/13410/hovercard" href="https://github.com/pandas-dev/pandas/issues/13410">#13410</a></p>
<p dir="auto">There's been discussion of a few overlapping uses of <code class="notranslate">Categorical</code>:</p>
<ol dir="auto">
<li>as 'true' categorical data with a known set of values</li>
<li>as 'lazy' categorical data which adds new categories as needed</li>
<li>as an interned string data type, with no particular categorical interpretation</li>
</ol>
<p dir="auto">Option 1 is currently well-supported. Option 2 can be achieved explicitly by <code class="notranslate">union_categorical</code>, see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="158501831" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/13361" data-hovercard-type="pull_request" data-hovercard-url="/pandas-dev/pandas/pull/13361/hovercard" href="https://github.com/pandas-dev/pandas/pull/13361">#13361</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="159399596" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/13410" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/13410/hovercard" href="https://github.com/pandas-dev/pandas/issues/13410">#13410</a>, but will not happen automatically e.g. when setting values or concatenating, see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="142872513" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/12699" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/12699/hovercard" href="https://github.com/pandas-dev/pandas/issues/12699">#12699</a>. Option 3 is similar to 2, and has been discussed as a new <code class="notranslate">String</code> type, see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="46855603" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/8640" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/8640/hovercard" href="https://github.com/pandas-dev/pandas/issues/8640">#8640</a>, but may have different semantics to 2.</p>
<p dir="auto">While I completely agree that option 1 should be the default, I'd like to see more support for option 2 if possible; there are cases where I really do want to work with categorical data, but I don't yet know what the categories are, e.g. when concatenating a table together from several different source files.</p>
<p dir="auto">One option would be to mimic Matlab's 'protected' flag for categorical data. By default, a <code class="notranslate">Categorical</code> would be created protected, and throw errors when performing actions which implicitly change the category set. However, the user could choose to declare a <code class="notranslate">Categorical</code> as unprotected, in which case these operations would be performed as intended.</p>
<p dir="auto">While this behaviour could be achieved with the proposed <code class="notranslate">String</code> type, it's unclear whether that type would share the <code class="notranslate">Categorical</code> API, or be efficiently convertible to <code class="notranslate">Categorical</code>. Having the behaviour as part of <code class="notranslate">Categorical</code> would allow the user to build up a <code class="notranslate">Categorical</code> iteratively, from multiple sources, and then quickly mark it protected once the category set is known.</p>
<p dir="auto">Currently, Categorical serves two main purposes</p>
<ol dir="auto">
<li>A type for expressing data from a fixed set of categories</li>
<li>A memory efficient storage format for low-cardinality objects</li>
</ol>
<p dir="auto">This proposal is to add a new extension type (let's call it <code class="notranslate">DictEncodedArray</code><br>
for now) for the second use case. The storage format would be the same as<br>
Categorical: an Index of the unique "keys" (categories) and an array of codes.<br>
Much of the implementation would be shared. But they would have different<br>
semantics on operations</p>
<ul dir="auto">
<li>concat (union by default)</li>
<li>groupby (unobserved categories would be dropped by default)</li>
<li>value_counts (unobserved categories would be dropped by default)</li>
</ul>
<p dir="auto">This is most useful for strings, but could even be useful for storing a large<br>
array of 64-bit precision items (store the 64-bit items once, then use an int16<br>
or int32 array for the codes).</p>