PERF: conversion to M8[ns]/datetime64[ns] is slow from "M8[s]" and other time resolutions
- [x ] I have checked that this issue has not already been reported.

- [x ] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

import pandas as pd, numpy as np, time
x = np.repeat(int(time.time()), 100_000_000)

"these are slow"
pd.Series(x.astype("M8[s]"))
pd.Series(x).astype("M8[s]")

"these are an order of magnitude faster"
pd.Series(x.astype("M8[s]").astype("M8[ns]"))
pd.Series((x*1e9).astype("M8[ns]")


#### Problem description

As long as there is no value translation involved converting/creating a datetime64[ns] array is fast as expected.  But it is an order of magnitude slower when there is a magnitude conversion.  Converting from float64 is only slightly slower than int64.

I've tested with 0.25.3, 0.25.32 and 1.0.5. And they all seem roughly consistent for those examples.  The older versions have added behavior that pd.Series((x*1e9).astype("M8[ns]"), dtype="datetime64[ns]") is also slow, but fast without the dtype arguement.

In all cases the result is the same series of datetime64[ns] values.

#### Output of ``pd.show_versions()``

python           : 3.8.3.final.0
python-bits      : 64
OS               : Linux
machine          : x86_64
processor        : x86_64
byteorder        : little
pandas           : 1.0.5
numpy            : 1.18.5
pytz             : 2020.1
dateutil         : 2.8.1