pd.to_datetime raises expection when converting date object with unit param
Hello, since pandas 0.18.1, converting to datetime a date defined as object, using the unit parameter, raises a ValueError exception.
#### Code Sample, a copy-pastable example if possible

```
import pandas as pd
import datetime

df = pd.DataFrame([[datetime.date(2016, 5, 10)]], index=[0], columns=['date'])

pd.to_datetime(df.date, unit='D')
```
#### Expected Output

Without the unit parameter, it does not raise the exception
#### output of `pd.show_versions()`
## INSTALLED VERSIONS

commit: None
python: 2.7.3.final.0
python-bits: 64
OS: Linux
OS-release: 3.5.0-37-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1
nose: 1.3.6
pip: None
setuptools: 5.1
Cython: None
numpy: 1.11.0
scipy: 0.17.0
statsmodels: 0.5.0
xarray: None
IPython: 3.1.0
sphinx: 1.2.2
patsy: 0.2.1
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: None
tables: 2.3.1
numexpr: 2.5
matplotlib: 1.4.3
openpyxl: 1.6.1
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: None
lxml: None
bs4: 4.3.2
html5lib: None
httplib2: 0.7.2
apiclient: None
sqlalchemy: 1.0.11
pymysql: None
psycopg2: None
jinja2: 2.7.2
boto: 2.27.0
pandas_datareader: 0.2.1

I've already tested with the master branch.
