Can't .loc[label] on a CategoricalIndex with labels being integer
I came around this and this seemed related to #9595 but could also be a bug in the above interacting with the CategoricalIndex. Using the same example as #15470 with pandas 0.20.3:

#### Code Sample
```python
    s = pd.Series([2, 1, 0], index=pd.CategoricalIndex([2, 1, 0]))
    s[2]  # works (interpreting as label)
    s.loc[2]  # fails with TypeError: cannot do label indexing on <class 'pandas.core.indexes.category.CategoricalIndex'> with these indexers [2] of <class 'int'>

    # of course the below works!
    s = pd.Series([2, 1, 0], index=[2, 1, 0])
    s[2]  # works (interpreting as label)
    s.loc[2]  # works (interpreting as label)
```

#### Problem description

`s.loc[2]` shouldn't fail and should select the label '2'.
Note this happens also with a pd.DataFrame.