Flaky xfail on master
e.g. https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=16270

```
=================================== FAILURES ===================================
_ TestCategoricalSeriesAnalytics.test_drop_duplicates_categorical_non_bool[True-datetime64[D]] _
[gw1] linux -- Python 3.7.4 /home/vsts/miniconda3/envs/pandas-dev/bin/python
[XPASS(strict)] GH#7996
_ TestCategoricalSeriesAnalytics.test_drop_duplicates_categorical_non_bool[False-datetime64[D]] _
[gw1] linux -- Python 3.7.4 /home/vsts/miniconda3/envs/pandas-dev/bin/python
[XPASS(strict)] GH#7996
_ TestCategoricalSeriesAnalytics.test_drop_duplicates_categorical_non_bool[None-datetime64[D]] _
[gw1] linux -- Python 3.7.4 /home/vsts/miniconda3/envs/pandas-dev/bin/python
[XPASS(strict)] GH#7996
```

xref https://github.com/pandas-dev/pandas/pull/27715.

Can you take a look @jbrockmendel?