BUG: df.sort_values w/ key function fails with multiple sort columns and Categorical sorting
- [ X] I have checked that this issue has not already been reported.

- [ X] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

#### Code Sample, a copy-pastable example

```python
import pandas as pd

df = pd.DataFrame(
    {
        "Name": ["baz", "foo", "baz", "foo", "bar", "bar", "bar"],
        "Criticality": ["High", "Critical", "Low", "Medium", "High", "Low", "Medium"],
    }
)

SEVERITY = ["Critical", "High", "Medium", "Low"]

def sorter(column):
    if column.name != "Criticality":
        return column
    cat = pd.Categorical(column, categories=SEVERITY, ordered=True)
    return pd.Series(cat)

df.sort_values(by=["Name", "Criticality"], key=sorter)
```

#### Problem description

This raises AttributeError: 'Series' object has no attribute 'categories'. If I sort by just Name, or just Criticality there are no issues. Likewise if I replace the sorter with the following it works sorting on both columns:

```python
def sorter(column):
    if column.name != "Criticality":
        return column
    mapper = {name: order for order, name in enumerate(SEVERITY)}
    return column.map(mapper)
```

#### Expected Output

I expected to be able to do a multi-column sort when one or more of them are Categorical, or at least being sorted categorically.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : 2a7d3326dee660824a8433ffd01065f8ac37f7d6
python           : 3.7.5.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 18.0.0
Version          : Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.1.2
numpy            : 1.18.2
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.2.3
setuptools       : 41.2.0
Cython           : None
pytest           : 5.4.1
hypothesis       : None
sphinx           : 2.4.4
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.5.0
html5lib         : None
pymysql          : 0.9.3
psycopg2         : 2.8.5 (dt dec pq3 ext lo64)
jinja2           : 2.11.1
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : 1.3.15
tables           : None
tabulate         : 0.8.7
xarray           : None
xlrd             : 1.2.0
xlwt             : None
numba            : None

</details>
