BUG: lib.infer_dtype croaking on Categoricals
```
In [13]: df = pd.DataFrame({'a':pd.Series(['A','B'], dtype='category')})
In [15]: lib.infer_dtype(df['a'])
TypeError: Cannot convert Categorical to numpy.ndarray
```
