REF: remove unnecessary _try_cast calls
Orthogonal to #29641.

_try_cast is a no-op if passed a scalar, so is unnecessary for some reduction/aggregation ops.  After #29641 i think we can get rid of more of these calls.