Median filter algorithm in opencv
I don't know about opencv's implementation, but a median filter can indeed be very slow because for every windows you have to sort your data in order to find the median. However there is a recursive implementation of the median filter, which is much more quick. It is very simple to build.

The basic idea is to recursively build the data's histogram that will allow you to very quickly get the median without sorting :
For the first window, fully build the histogram, then, shift your windows by one step (or more, it depends on you), remove the last value from the previous window and add the new one, you updated your histogram with only 2 operations.
Then you have to find the median within the histogram, this is recursively feasible but I don't remember the exact implementation (it is a bit more tricky), however simply checking the histogram's cumulative sum, while brutal, will give you the median quickly (worst case scenario is N summation, with N your window's size).