BUG: Segmentation fault when doing pandas.core.window.rolling.RollingGroupBy.apply
- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example

```python
import pandas as pd

df = pd.DataFrame(
    [
        ["A", "group_1", pd.Timestamp(2019, 1, 1, 9)],
        ["B", "group_1", pd.Timestamp(2019, 1, 2, 9)],
        ["C", "group_2", pd.Timestamp(2019, 1, 3, 9)],
        ["D", "group_1", pd.Timestamp(2019, 1, 6, 9)],
        ["E", "group_1", pd.Timestamp(2019, 1, 7, 9)],
        ["F", "group_1", pd.Timestamp(2019, 1, 10, 9)],
        ["G", "group_2", pd.Timestamp(2019, 1, 20, 9)],
        ["H", "group_1", pd.Timestamp(2019, 4, 8, 9)],
    ],
    columns=["index", "group", "eventTime"],
).set_index("index")

groups = df.groupby("group")
df["count_to_date"] = groups.cumcount()
rolling_groups = groups.rolling("10d", on="eventTime")
group_size = rolling_groups.apply(lambda df: df.shape[0])
print(group_size)
```

#### Problem description
The above code causes a segmentation fault inside pandas for versions *after* 1.0.5. Since I need the above code for a project, I am restricted to using pandas 1.0.5 until this is resolved. I am not sure what is causing the segmentation fault, but all the above circumstances are necessary to reproducing the bug (ie `DataFrame` with special index, a column set in the `DataFrame` after grouping, a rolling window on a group, etc).

I have reproduced this bug on a variety of machines and operating systems.

#### Expected Output

```
                        eventTime  count_to_date
group   index                                   
group_1 A     2019-01-01 09:00:00            1.0
        B     2019-01-02 09:00:00            2.0
        D     2019-01-06 09:00:00            3.0
        E     2019-01-07 09:00:00            4.0
        F     2019-01-10 09:00:00            5.0
        H     2019-04-08 09:00:00            1.0
group_2 C     2019-01-03 09:00:00            1.0
        G     2019-01-20 09:00:00            1.0
```

Note: This is indeed the output of versions 1.0.5 and prior.

#### Output of ``pd.show_versions()``
This is just one configuration but the bug has been reproduced on three different machines (both linux and mac), all exhibiting the same behavior.

<details>

INSTALLED VERSIONS
------------------
commit           : 2a7d3326dee660824a8433ffd01065f8ac37f7d6
python           : 3.7.3.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 17.7.0
Version          : Darwin Kernel Version 17.7.0: Thu Jun 18 21:21:34 PDT 2020; root:xnu-4570.71.82.5~1/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.1.2
numpy            : 1.18.2
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 40.8.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.2.1
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : 1.5.2
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None
</details>
