BUG: groupby nunique changing values
- [ ] closes #31950 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

just reopening #31973 for now - this doesn't special-case NaT. Will think if there's a better way to do it, feel free to close / take over if it's wrong