BUG: pd.Series.mode fails (sometimes) as aggregation method on groupby object
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

#### Code Sample, a copy-pastable example

This popped up randomly for some of my dataframes, but only a few.
EDIT: Thanks to @mzeitlin11 below for a properly minimal example. 

```python
import pandas as pd

df = pd.DataFrame([1, 1, 2, 2])
df.groupby([0, 0, 0, 0]).agg([pd.Series.mode])
```

</details>

This results in a `ValueError: no results`:

<details>

```python
ValueError                                Traceback (most recent call last)
c:\tmp\zonal-mode\debug.py in 
----> 7 selection.groupby("id").agg([pd.Series.mode])

C:\Miniconda3\envs\main\lib\site-packages\pandas\core\groupby\generic.py in aggregate(self, func, engine, engine_kwargs, *args, **kwargs)
    943         func = maybe_mangle_lambdas(func)
    944 
--> 945         result, how = aggregate(self, func, *args, **kwargs)
    946         if how is None:
    947             return result

C:\Miniconda3\envs\main\lib\site-packages\pandas\core\aggregation.py in aggregate(obj, arg, *args, **kwargs)
    584         # we require a list, but not an 'str'
    585         arg = cast(List[AggFuncTypeBase], arg)
--> 586         return agg_list_like(obj, arg, _axis=_axis), None
    587     else:
    588         result = None

C:\Miniconda3\envs\main\lib\site-packages\pandas\core\aggregation.py in agg_list_like(obj, arg, _axis)
    672         raise ValueError("no results")
    673 
--> 674     try:
    675         return concat(results, keys=keys, axis=1, sort=False)
    676     except TypeError as err:

ValueError: no results
```

</details>


other methods seem to work okay:

```python
selection.groupby("id").agg([pd.Series.sum])
selection.groupby("id").agg([pd.Series.mean])
selection.groupby("id").agg([pd.Series.std])
selection.groupby("id").agg(["count"])
```

Oddly enough, slightly changing the selection makes it run:

```python
selection.iloc[100:].groupby("id").agg([pd.Series.mode])  # works
selection.iloc[:900].groupby("id").agg([pd.Series.mode])  # works
selection.iloc[:917].groupby("id").agg([pd.Series.mode])  # works
selection.iloc[:918].groupby("id").agg([pd.Series.mode])  # errors
selection.iloc[18:918].groupby("id").agg([pd.Series.mode])  # works
selection.iloc[17:918].groupby("id").agg([pd.Series.mode])  # errors
```

`scipy.stats.mode` also works fine:

```python
import scipy.stats
selection.groupby("id").agg([scipy.stats.mode])
```

#### Problem description

I don't understand why it seems to randomly fail, there are no oddities with the data as far as I can tell.
Given the difference the selections make, I guess it's related with the specific number of occurences, which would suggest something internally in implementation of pd.Series.mode.

#### Expected Output

#### Output of ``pd.show_versions()``

<details>


INSTALLED VERSIONS
------------------
commit           : 2cb96529396d93b46abab7bbc73a208e708c642e
python           : 3.8.8.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.18362
machine          : AMD64
processor        : Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : Dutch_Netherlands.1252

pandas           : 1.2.4
numpy            : 1.20.2
pytz             : 2021.1
dateutil         : 2.8.1
pip              : 21.1.1
setuptools       : 49.6.0.post20210108
Cython           : None
pytest           : 6.2.4
hypothesis       : None
sphinx           : 3.5.4
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.3
IPython          : 7.23.1
pandas_datareader: None
bs4              : None
bottleneck       : 1.3.2
fsspec           : 2021.04.0
fastparquet      : None
gcsfs            : None
matplotlib       : 3.4.1
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : 1.6.3
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : 0.18.0
xlrd             : None
xlwt             : None
numba            : 0.52.0

</details>
