Exception when grouping on a modified category
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd

ds = pd.DataFrame({"a": [1, 2, 3], "b": ["a", "b", "c"]}) 
ds["b"] = ds["b"].astype("category")
ds['b'].cat.remove_categories(['a'], inplace=True)

# works
ds.groupby("b").groups

# gives ValueError: Categorial categories cannot be null 
ds.groupby("b", observed=True).groups

```
#### Problem description

I wonder if that exception should be raised there?
Using 0.24 RC1

Thanks to all working on pandas!