AssertionError when grouping with max/min as aggregation functions (pandas-1.0.0)
#### Code Sample

```python
import pandas as pd
import numpy as np

df = pd.DataFrame({'key1' : ['a', 'a', 'b', 'b', 'a'],
                   'key2' : ['one', 'two', 'one', 'two', 'one'],
                   'key3' : ['three', 'three', 'three', 'six', 'six'],
                   'data1' : np.random.randn(5),
                   'data2' : np.random.randn(5)})
df.groupby('key1').min()
```
#### Problem description

Since `pandas-1.0.0`, an `AssertionError` is thrown when grouping a `DataFrame` by a key and using `max`/`min` as aggregation functions. It works fine if only 1 key (other than the grouping key) is of the type `object` in the DataFrame, but it doesn't when the number of keys of type `object` is bigger than 1 (as shown in the example). This configuration worked fine on previous versions of `pandas` (e.g., `pandas-0.25.3`).

#### Expected Output

| key1   | key2   | key3   |    data1 |     data2 |
|:-------|:-------|:-------|---------:|----------:|
| a      | one    | six    | -0.67246 | -1.6302   |
| b      | one    | six    | -1.72628 | -0.907298 |

#### Output of ``pd.show_versions()``

commit           : None
python           : 3.7.6.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 19.2.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.UTF-8

pandas           : 1.0.0
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 45.1.0.post20200127
Cython           : 0.29.14
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.11.1
pandas_datareader: None
bs4              : None
bottleneck       : 1.3.1
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : None
pyxlsb           : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : None
tables           : None
tabulate         : 0.8.3
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : 0.48.0