BUG: GroupBy.mean() is extremely slow with sparse arrays
- [x ] I have checked that this issue has not already been reported.

- [ x] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

#### Code Sample, a copy-pastable example

```python
import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame(np.random.rand(1000, 1000) > 0.8, dtype=int)
df['g'] = ['A' if x else 'B' for x in np.random.rand(1000) > 0.5]

%timeit -n 3 -r 5 df.groupby('g').mean()

sdf = df.set_index('g').astype('Sparse[int]').reset_index()
%timeit -n 3 -r 5 sdf.groupby('g').mean()
```

Output:

Dense:
`11 ms ± 430 µs per loop (mean ± std. dev. of 5 runs, 3 loops each)`

Sparse:
`3.91 s ± 33.5 ms per loop (mean ± std. dev. of 5 runs, 3 loops each)`

or subset of a real-word dataset:

```python
import pandas as pd

df = pd.read_csv('https://github.com/pandas-dev/pandas/files/5176354/pbmc-sparse-df.csv.gz', 
                 index_col=0).astype(int).reset_index()
%timeit -n 3 -r 5 df.groupby('Cell type').mean()

sdf = df.set_index('Cell type').astype(pd.SparseDtype(int, fill_value=0)).reset_index()
%timeit -n 3 -r 5 sdf.groupby('Cell type').mean()
```

Output:

Dense:
`9.2 ms ± 403 µs per loop (mean ± std. dev. of 5 runs, 3 loops each)`

Sparse:
`3.73 s ± 126 ms per loop (mean ± std. dev. of 5 runs, 3 loops each)`


#### Problem description

`GroupBy.mean()` with the SparseArray seems extremely slow compared to dense matrices (here around 355 times).

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : f2ca0a2665b2d169c97de87b8e778dbed86aea07
python           : 3.8.5.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 19.6.0
Version          : Darwin Kernel Version 19.6.0: Thu Jun 18 20:49:00 PDT 2020; root:xnu-6153.141.1~1/RELEASE_X86_64
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.1.1
numpy            : 1.19.1
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 20.2.1
setuptools       : 49.2.1.post20200802
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.17.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : 0.8.0
fastparquet      : None
gcsfs            : None
matplotlib       : 3.3.0
numexpr          : 2.7.1
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 1.0.0
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : 1.5.2
sqlalchemy       : None
tables           : 3.6.1
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : 0.50.1

</details>

I attached the file here for reproducibility.

[pbmc-sparse-df.csv.gz](https://github.com/pandas-dev/pandas/files/5176354/pbmc-sparse-df.csv.gz)
