BUG: Segmentation-fault during a "custom" rolling operation
### 

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the [latest version](https://pandas.pydata.org/docs/whatsnew/index.html) of pandas.

- [ ] I have confirmed this bug exists on the master branch of pandas.


### Reproducible Example

```python
from pandas.core.window.indexers import BaseIndexer
from typing import Union, Tuple, Optional

import numpy as np
import pandas as pd


class RollingFixedOffsetVariableWindowIndexer(BaseIndexer):

    def get_window_bounds(self, num_values: int = 0, min_periods: Optional[int] = None, center: Optional[bool] = None,
                          closed: Optional[str] = None) -> Tuple[np.ndarray, np.ndarray]:
        # Every hour
        start = np.arange(0, num_values, 60 * 60)
        # Every hour + day
        end = start + 60 * 60 * 24

        return start, end


ser = pd.Series(np.arange(100000), index=pd.date_range(start='2020', freq='1s', periods=100000))
idxr = RollingFixedOffsetVariableWindowIndexer(window_size=60 * 60 * 24)
out = ser.rolling(idxr, min_periods=10).mean()
```


### Issue Description

(Maybe related to #43267 ? Couldn't tell.)

I'm trying to create a custom `BaseIndexer` for a `.rolling` operation. The extraction of the window is quick and efficient, yet when I attempt to perform various operations on it I get the seg-fault error:
> Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

I have created an MWE for this behavior using a fixed-indexer (that carries the same name). The original indexer was supposed to produce a rolling-operation with overlaps, so the above indexer is similar in the sense that it creates a 1-day windows 1-hour apart for a one-second index.

### Expected Behavior

Hmmm... Not to have a seg-fault error?

### Installed Versions

INSTALLED VERSIONS
------------------
commit           : 945c9ed766a61c7d2c0a7cbb251b6edebf9cb7d5
python           : 3.8.9.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.4.0-77-generic
Version          : #86~18.04.1-Ubuntu SMP Fri Jun 18 01:23:22 UTC 2021
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.3.4
numpy            : 1.18.5
pytz             : 2018.3
dateutil         : 2.8.1
pip              : 21.1.3
setuptools       : 57.4.0
Cython           : None
pytest           : 6.2.2
hypothesis       : None
sphinx           : 3.5.3
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : 0.999999999
pymysql          : None
psycopg2         : None
jinja2           : 2.11.3
IPython          : None
pandas_datareader: None
bs4              : 4.6.0
bottleneck       : None
fsspec           : 0.8.7
fastparquet      : None
gcsfs            : None
matplotlib       : 3.1.3
numexpr          : 2.7.3
odfpy            : None
openpyxl         : 3.0.7
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : None
tables           : 3.6.1
tabulate         : 0.8.9
xarray           : None
xlrd             : 1.2.0
xlwt             : None
numba            : None
None

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
