Remove properties from groupby whitelists
This is a small piece broken off of #17179, keeping a modicum of forward momentum going.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
