groupby observed=True not working for aggregating a column
In the following the parameter `observed=True` will be ineffective (in pandas 0.23.4)
```python
import pandas as pd
dd=pd.DataFrame({"c":pd.Series([1,1,2], dtype="category"), "i":[1,2,2], "x":[1, 2, 3]})
gr=dd.groupby(["c", "i"], as_index=False, observed=True)["x"].sum()
gr   # will contain a NaN for c=2, i=1
```
It will only work as expected if I aggregate over all DataFrame columns, i.e. leaving out `["x"]`.

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.4
pytest: 3.10.1
pip: 18.1
setuptools: 40.5.0
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: None
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.8
feather: None
matplotlib: 3.0.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>
