REGR: to_datetime, unique with OOB values
Closes https://github.com/pandas-dev/pandas/issues/31491

This turned up two bugs:

1. `to_datetime(oob_values, cache=True, errors='coerce')` would raise. We had tests intended to catch this, but we didn't actually hit it because caching is disabled for inputs less than length 50. We only tested with length 3. I've updated the test to pass that threshold
2. The original report, that `to_datetime` was raising for non-ns resolution values that aren't OOB for ns resolution. This was a bug in `unique`. 