Replacing mean by median over batch-size to lessen the impact of outliers
One shortcoming is that the median is often more computational expensive to calculate than the mean. The median can be calculated with a variation of quickselect which is linear worst-case performance. Calculating the mean only requires the sum and count of the numbers.