PERF: non-info axes slicing on Panels is slow
Assume we have two panels:

```
>>> pn1.shape
(13, 5412, 162)
>>> pn2.shape
(12, 5412, 162)
```

12 fields are float64 while one field in `pn1` is `datetime64[ns]`. This slows pretty much all operations (slicing, querying, anything) down by a huge factor:

```
>>> %timeit pn2.values
100000 loops, best of 3: 4.06 µs per loop

>>> %timeit pn1.values
1 loops, best of 3: 5.31 s per loop
```

Is there an unofficial rule of not using datetime64 in the first place, is it some weird coercion bug (it seems to try and coerce everything down to floats), does it have anything to do with panels?
