BUG: date is parsed wrong
- [x] I have checked that this issue has not already been reported.

- [x] I have confirmed this bug exists on the latest version of pandas.


#### Code Sample, a copy-pastable example
**1.** first I type in the date in Ms excel's B2 cell in local language format, as you can read from the picture ![1](https://user-images.githubusercontent.com/7960913/80488974-c238ff80-8991-11ea-9d21-41bcac1dfd76.png) that it is been recognized as date(2020/12/20) without any problem. Then I save it as [`test.xlsx`](https://github.com/pandas-dev/pandas/files/4545801/test.xlsx) . Then I try to read the date info in pandas

```python
# I have tested both openpyxl and xlrd. All of them performs alike
# df = pd.read_excel('test.xlsx', engine='openpyxl')   
df = pd.read_excel('test.xlsx', engine='xlrd')
print(df.loc[0, 'date'])   # which outputs `44185`
```

#### Problem description
I can't fetch the right date in any of the following 3 methods

`pd.to_datetime(df['date'])` outputs 
```
0   1970-01-01 00:00:00.000044185
Name: date, dtype: datetime64[ns]
```

`pd.to_datetime(df['date'], format='%Y/%m/%d')` outputs
```
0   1970-01-01 00:00:00.000044185
Name: date, dtype: datetime64[ns]
```

and
`pd.to_datetime(df['date'], unit='D')` outputs
```
0   2090-12-22
Name: date, dtype: datetime64[ns]
```

#### Expected Output

2020-12-20, i.e. Dec 20th, 2020
 
**2.** however If I change the cell format to `m/d/yyyy`, then change it back to custom-defined local language format `yyyy"年"m"月"d"日"` and save the file as [`test2.xlsx`](https://github.com/pandas-dev/pandas/files/4545802/test2.xlsx). Then 
```python
import pandas as pd
# I have tested both openpyxl and xlrd. All of them performs alike
df = pd.read_excel('test2.xlsx', engine='openpyxl')
# df = pd.read_excel('test2.xlsx', engine='xlrd')
print(df.loc[0, 'date'])
```
outputs `2020-12-20 00:00:00` as expected

**3.** I can't tell is this a bug/feature in xlrd/openpyxl or pandas

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.6.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 1.0.3
numpy            : 1.18.2
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 41.2.0
Cython           : 0.27.3
pytest           : 3.4.2
hypothesis       : None
sphinx           : 1.7.1
blosc            : None
feather          : None
xlsxwriter       : 1.0.2
lxml.etree       : 4.2.2
html5lib         : 0.9999999
pymysql          : None
psycopg2         : None
jinja2           : 2.10
IPython          : 6.5.0
pandas_datareader: None
bs4              : 4.6.0
bottleneck       : 1.2.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.2.2
matplotlib       : 2.2.2
numexpr          : 2.6.4
odfpy            : None
openpyxl         : 3.0.3
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 3.4.2
pyxlsb           : None
s3fs             : None
scipy            : 1.0.0
sqlalchemy       : 1.2.4
tables           : 3.4.4
tabulate         : None
xarray           : None
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.0.2
numba            : 0.37.0

</details>
