Merging Pandas DataFrames on categorical series
Question: I'm trying to understand if pandas supports merging DataFrames on columns of
categorical data (i.e. dtype="category").

I do most of my data work in R, but am trying to do more work in
Python/pandas. In R, merging on factors (analogous to the categorical dtype)
induces type coercion, typically to character. This allows one data frame to
have a by-variable (join column) specified as a factor (categorical) and the
other to have its by-variable be a string. Does pandas perform similar
coercion of categorical data to string prior to merging/joining? Should I
expect merging on categoricals to be robust? Where can I find documentation on
(automatic) type coercion in pandas?

Simple example:

+++ It is an error to test a categorical vector for equality against a non-
categorical/non-scalar vector:

    
    
    In [52]:
    import pandas as pd
    
    a = pd.Series(['a','b','c'],dtype="category")
    b = pd.Series(['a','b','c'],dtype="object")
    c = pd.Series(['a','b','cc'],dtype="object")
    
    In [54]:
    
    a==b
    
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    ...
    TypeError: Cannot compare a Categorical for op <built-in function eq> with type <class         'numpy.ndarray'>. If you want to compare values, use 'series <op> np.asarray(cat)'.
    

+++ But merging a DataFrame on columns of different type--one categorical, one
string--does not throw an error (at least in this simple case). Some type of
coercion must occur:

    
    
    In [59]:
    A = pd.DataFrame({'A':a,'B':[1,2,3]})
    B = pd.DataFrame({'A':b,'C':[4,5,6]})
    print(A.merge(B,on='A'))
       A  B  C
    0  a  1  4
    1  b  2  5
    2  c  3  6
    


Answer: So in short, in 0.15.1 the merging behavior was changed (fixed really) to
allow merging of Categoricals that had exactly the same categories. Further if
an object array was merged in it is allowed, but the resulting character of
the returned merge would now be object (IIRC). I don't recall if we try to
infer it back to a Categorical or not.

I created an issue [here](https://github.com/pydata/pandas/issues/8938) for
discussion on this.

The equality shown above, e.g. not allowing comparisons of Categoricals vs
object dtypes was done first, while the merging behavior was recently expanded
to allow the merging of like-Categoricals and objects dtypes (assuming all
merged Categoricals share the same categories).

So I think allowing the equality to work is just the API not catching up. We
will address this in 0.16.0, but pls provide comments on the issue.

PR for this is [here](https://github.com/pydata/pandas/pull/8946)

This will be in the upcoming 0.15.2 release (slated for week of December 7,
2014)

