<p dir="auto">Should pandas have a  <em>Symbol</em> datatype for strings which are frequently re-used?</p>
<p dir="auto">For example, consider a table of stock trades with columns ['Timestamp','Price','Size','StockSymbol']. Suppose there are 10 million trades, but only 500 different stock symbols. Instead of storing 10 million strings, we could fill the 'StockSymbol' column with integers 0-499 where 0 represents AAPL, 1 represents AMZN, etc.</p>
<p dir="auto">The <em>Symbol</em> type in Ruby, Scala, and Q does this compression/expansion automatically. It's a big memory saver in big tables, and <em>Symbol</em> comparison can be much faster than string comparison.</p>
<p dir="auto">I often use <code class="notranslate">.astype('category')</code> for string compression in pandas. But at a talk I saw recently, <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wesm/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wesm">@wesm</a> suggested this might not be a great idea. I've definitely caused several confusing problems by abusing categoricals this way: comparing two columns with different levels, appending rows with new symbols, wrongly assuming <code class="notranslate">sort_values()</code> will sort symbols lexicographically, etc.</p>
<p dir="auto">Related question: Without a <em>Symbol</em> datatype, is there a consensus on best practice for storing repetitive strings in a Series or DataFrame? Should we leave them as <code class="notranslate">Object</code> type? Compress them as un-ordered categoricals? Do something else I haven't thought of?</p>
<p dir="auto"><em>update for 2019-10-07: We have a StringDtype extension dtype. It's memory model is the same as the old implementation, an object-dtype ndarray of strings. The next step is to store &amp; process it natively</em>.</p>
<hr>
<p dir="auto">xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="46780098" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/8627" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/8627/hovercard" href="https://github.com/pandas-dev/pandas/issues/8627">#8627</a><br>
xref <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="46859261" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/8643" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/8643/hovercard" href="https://github.com/pandas-dev/pandas/issues/8643">#8643</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="43427084" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/8350" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/8350/hovercard" href="https://github.com/pandas-dev/pandas/issues/8350">#8350</a></p>
<p dir="auto">Since we introduced <code class="notranslate">Categorical</code> in 0.15.0, I think we have found 2 main uses.</p>
<ol dir="auto">
<li>as a 'real' Categorical/Factor type to represent a limited of subset of values that the column can take on</li>
<li>as a memory saving representation for object dtypes.</li>
</ol>
<p dir="auto">I could see introducting a <code class="notranslate">dtype='string'</code> where <code class="notranslate">String</code> is a slightly specialized sub-class of <code class="notranslate">Categroical</code>, with 2 differences compared to a 'regular' Categorical:</p>
<ul dir="auto">
<li>it allows unions of arbitrary other string types, currently <code class="notranslate">Categorical</code> will complain if you do this:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="In [1]: df = DataFrame({'A' : Series(list('abc'),dtype='category')})
In [2]: df2 = DataFrame({'A' : Series(list('abd'),dtype='category')})
In [3]: pd.concat([df,df2])
ValueError: incompatible levels in categorical block merge"><pre class="notranslate"><code class="notranslate">In [1]: df = DataFrame({'A' : Series(list('abc'),dtype='category')})
In [2]: df2 = DataFrame({'A' : Series(list('abd'),dtype='category')})
In [3]: pd.concat([df,df2])
ValueError: incompatible levels in categorical block merge
</code></pre></div>
<p dir="auto">Note that this works if they are <code class="notranslate">Series</code> (and prob should raise as well, side -issue)</p>
<p dir="auto">But, if these were both 'string' dtypes, then its a simple matter to combine (efficiently).</p>
<ul dir="auto">
<li>you can restrict the 'sub-dtype' (e.g. the dtype of the categories) to <code class="notranslate">string/unicode</code> (iow, don't allow numbers / arbitrary objects), makes the constructor a bit simpler, but more importantly, you now have a 'real' non-object string dtype.</li>
</ul>
<p dir="auto">I don't think this would be that complicated to do. The big change here would be to essentially convert any object dtypes that are strings to <code class="notranslate">dtype='string'</code> e.g. on reading/conversion/etc. might be a perf issue for some things, but I think the memory savings greatly outweigh.</p>
<p dir="auto">We would then have a 'real' looking object dtype (and <code class="notranslate">object</code> would be relegated to actual python object types, so would be used much less).</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/shoyer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/shoyer">@shoyer</a><br>
cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/JanSchulz/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/JanSchulz">@JanSchulz</a><br>
cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jorisvandenbossche/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jorisvandenbossche">@jorisvandenbossche</a><br>
cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mwiebe/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mwiebe">@mwiebe</a><br>
thoughts?</p>