pivot_table very slow on Categorical data; how about an observed keyword argument? #24923
- [x] closes #24923
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
