setting a whole column changes column dtype from "category" to "object"
#### Code Sample, a copy-pastable example if possible

```python
# Your code here
import pandas as pd
from enum import Enum, IntEnum, auto

# Your code here
class ConnectionRoles(Enum):
    Client = auto()
    Server = auto()

dtype_role = pd.api.types.CategoricalDtype(categories=list(ConnectionRoles), ordered=True)

df  = pd.DataFrame(columns=["tcpdest"])

df = df.astype(dtype= { "tcpdest": dtype_role })

print("before setting ", df.dtypes.tcpdest)
df['tcpdest'] = ConnectionRoles.Server

print("after setting", df.dtypes.tcpdest)

```
#### Problem description
Setting the whole serie changes the dtype from category to object which breaks my code afterwards.

#### Expected Output
before setting  category
after setting category

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
Details

ommit: None
python: 3.6.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.24
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8
LOCALE: fr_FR.UTF-8

pandas: 0.23.3
pytest: None
pip: 18.0
setuptools: 40.0.0
Cython: 0.28.3
numpy: 1.14.5
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.6
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.4
xlrd: 0.9.4
xlwt: 1.3.0
xlsxwriter: None
lxml: 4.2.3
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.10
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>
