DataFrame.groupby.sum() is extremely slow when dtype is timedelta64[ns] compared to int64.
#### Steps to demonstrate

```python
import numpy as np
import pandas as pd

td = pd.DataFrame(np.random.randint(1000000, 100000000, (10000,100)), index=np.random.randint(200, size=(10000,))).astype('timedelta64[ns]')
i = td.copy(deep=True).astype('int64')

%time temp1 = td.groupby(lambda x: x).sum() # 2.33 s
%time temp2 = i.groupby(lambda x: x).sum() # 15.6 ms

temp2 = temp2.astype('timedelta64[ns]')
assert((temp1 == temp2).values.all())
```
#### Problem description

When performing a summation on grouped 'timedelta64[ns]' data, there is a significant performance decrease compared to the same data interpreted as 'int64'.

Possibly related to https://github.com/pandas-dev/pandas/issues/18053

#### Expected Behavior

It is my understanding that internally 'timedelta64[ns]' are just 'int64' and are interpreted as a count of 'ns'. Shouldn't the summation performance be equal in that case?

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.22.0
pytest: 3.5.0
pip: 9.0.1
setuptools: 39.0.1
Cython: 0.28.1
numpy: 1.14.2
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.7.2
patsy: 0.5.0
dateutil: 2.7.2
pytz: 2018.3
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.1
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.5
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>
