Median filtering (MF) is a canonical image processing operation truly useful in many practical applications. The MF most appealing feature is its resistance to noise and errors in data, but because the method requires window values to be sorted it is computationally expensive. In this work, a new insight into MF capabilities based on the optimal breakdown value (BV) of the median is offered, and it is also shown that the BV-based versions of two of the most popular MF algorithms outperform their corresponding standard versions. A general framework for both the theoretical analysis and comparison of MF algorithms is presented in the process, which will hopefully contribute to a better understanding of the MF many subtle features. The introduced ideas are experimentally tested by using real and synthetic images.