Data dependent bug in mode aggregation
#### Code Sample, a copy-pastable example if possible

The first line is the problem, the others given for context.  My goal in the function this comes from is to remove non-modal ("atypical") items per `key`.  The task should, and does, allow multi-modal targets per group (there are other techniques to pick a unique, but arbitrary mode, but I want all of them).

```python
# Your code here
modes = pd.DataFrame(df.groupby('key').value.agg(pd.Series.mode))
df.set_index('key', inplace=True)
df = df.join(modes, rsuffix='_mode')
df = df[df.apply(lambda row:row.value in row.value_mode, axis=1)]
```

The below uses a DataFrame smaller than the one where the problem was encountered.  As discussed in a comment below, the precise sampling of the original data and the precise anonymization of groupby keys used has an unpredictable effect on the problem being accidentally fixed or continuing to occur.

```python
>>> import pandas as pd
>>> import gzip
>>> import pickle
>>> df = pickle.load(gzip.open('modal-crash.pkl.gz'))
>>> modes = df.groupby('key').value.agg(pd.Series.mode)
Traceback (most recent call last):
  File "<ipython-input-13-4fe1f0dd3977>", line 1, in <module>
    modes = df.groupby('key').value.agg(pd.Series.mode)
  File "/anaconda3/envs/boldmetrics-analysis/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 778, in aggregate
    result = self._aggregate_named(func_or_funcs, *args, **kwargs)
  File "/anaconda3/envs/boldmetrics-analysis/lib/python3.7/site-packages/pandas/core/groupby/generic.py", line 909, in _aggregate_named
    raise Exception('Must produce aggregated value')
Exception: Must produce aggregated value

```

[modal-crash.pkl.gz](https://github.com/pandas-dev/pandas/files/2923871/modal-crash.pkl.gz)

#### Problem description

In a manner sensitive to the exact data, but not following any obvious pattern I have discerned, the `modes = ...` line can produce 3-4 levels of nested exceptions leading to `Exception: Must produce aggregated value`.

I have checked the "problem" data to make sure there are no missing values in `df.value`.  All values are present and are an expected (string) value.  This particular task has to do with a target that is a clothing size.  E.g. when run correctly, `modes` will resemble this:

```
             value
key
ddhhmttmvv   [L, M]
dfdavpqthh      [S]
ddfadjqqhd      [M]
ddvjmatfvm      [M]
... 160k more rows ...
```

The code succeeds for **most** initial DataFrames.  In fact, adding `df = df.iloc[:-1]` or `df = df.iloc[1:]` makes the operation succeed for every "problem" DataFrame I have encountered.

I have identified a hack/fix to the problem, but it's ad hoc and ugly:

```python
def mode(s):
    return list(pd.Series.mode(s))
modes = pd.DataFrame(df.groupby('key').value.agg(mode))
```

#### Expected Output

Expected output shown in the problem description.  The DataFrame generated after the filtering is just a subset of rows in the original, but the problem only occurs in finding the mode per group.

#### Output of ``pd.show_versions()``

<details>
% python -c "import pandas; pandas.show_versions()"
/anaconda3/envs/boldmetrics-analysis/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.1
pytest: None
pip: 19.0.3
setuptools: 40.8.0
Cython: 0.29.5
numpy: 1.16.1
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: 7.3.0
sphinx: 1.8.4
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: 2.6.9
feather: None
matplotlib: 3.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: 4.3.1
bs4: None
html5lib: None
sqlalchemy: 1.2.18
pymysql: None
psycopg2: 2.7.7 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
</details>
