BUG: output of a transform is cast to dtype of input
xref #11444, #13046 for addtl tests

``` python
In [27]: df = pd.DataFrame({'a': np.random.randint(0, 5, 365), 'b': pd.date_range('2015-01-01', periods=365, freq='D')})

In [28]: df.head()
Out[28]:
   a          b
0  3 2015-01-01
1  3 2015-01-02
2  4 2015-01-03
3  2 2015-01-04
4  4 2015-01-05

In [29]: df.groupby('a').b.transform(lambda x: x.dt.dayofweek - x.dt.dayofweek.mean()).head()
Out[29]:
0   1970-01-01 00:00:00.000000000
1   1970-01-01 00:00:00.000000001
2   1970-01-01 00:00:00.000000001
3   1970-01-01 00:00:00.000000002
4   1969-12-31 23:59:59.999999997
Name: b, dtype: datetime64[ns]
```

I expected a float. No idea how difficult this will be so I marked it for 0.18. I won't have time to get to it any earlier, but if someone else wants to...
