BUG: Categorical RollingGroupby will raise IndexError
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---


```python
import pandas as pd
print(pd.__version__, '\n')

df = pd.DataFrame([1, 2, 3, 4, 5],
                  index=pd.CategoricalIndex(["a", "b", "b", "a", "c"]))

print(df)
print(df.iloc[:4].groupby(level=0).rolling(2).sum())
```

#### Problem description

When the index contains only part of the Categorical, RollingGroupby will raise an error:

```
pandas\core\window\indexers.py in get_window_bounds(self, num_values, min_periods, center, closed)
    320             # Extend as we'll be slicing window like [start, end)
    321             window_indicies = np.append(
--> 322                 window_indicies, [window_indicies[-1] + 1]
    323             ).astype(np.int64)
    324             start_arrays.append(window_indicies.take(start))

IndexError: index -1 is out of bounds for axis 0 with size 0
```

This problem has appeared since 1.1.0, and 1.0.5 runs normally

#### Expected Output

```
1.0.5 

   0
a  1
b  2
b  3
a  4
c  5
       0
a a  NaN
  a  5.0
b b  NaN
  b  5.0
```

<details>

INSTALLED VERSIONS
------------------
commit           : db08276bc116c438d3fdee492026f8223584c477
python           : 3.7.9.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.19041
machine          : AMD64
processor        : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 1.1.3
numpy            : 1.19.2
pytz             : 2020.4
dateutil         : 2.8.1
pip              : 20.3.3
setuptools       : 51.0.0.post20201207
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.6.2
html5lib         : 1.1
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.19.0
pandas_datareader: None
bs4              : 4.9.3
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.3.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.15.1
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : 1.5.2
sqlalchemy       : None
tables           : None
tabulate         : 0.8.7
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

</details>
