PERF: Avoid materializing values in `Categorical.set_categories`
In `Categorical.set_categories`, we allocate an array of the values, which may be expensive: https://github.com/pandas-dev/pandas/blob/83436af8ae1ccad49b7ceac7471c060d823d10ab/pandas/core/categorical.py#L774

It should be possible to do this operation by just manipulating the codes.

```python
In [6]: c = pd.Categorical(['a'] * 100000)

In [7]: c.set_categories(['a', 'b'])
Out[7]:
[a, a, a, a, a, ..., a, a, a, a, a]
Length: 100000
Categories (2, object): [a, b]

```

See https://github.com/pandas-dev/pandas/pull/16015/commits/5ab01235110f706227651d5303e5e9ed060b05cd for how this might work, which will probably be squashed, but it's the implementation of `Categorical._set_dtype` in https://github.com/pandas-dev/pandas/pull/16015/

I may get to this as a followup to that PR.