Fix performance regression with Series operations
#25743 caused a performance regression with Series operations (mean, sum, etc).  This can be avoided by moving some of the `if`s around so that the additional check is not performed for those operations.