REF: Pre-empt ValueError in _aggregate_series_fast
cc @WillAyd @jreback orthogonal to other groupby PRs (including #29499) but will have merge conflicts.  No preference on what order they should go in.