Design a filter that allows frequencies below 5 kHz to pass freely, but all frequencies above 5.2 kHz must be undetectable
You have assumed a 20dB/dec per filter order roll-off for your filter. This is not true for all filter types.

Let \$f_0 = 5 \mathrm{kHz}\$ and \$f_{\mathrm{stop}} = 5.2 \mathrm{kHz}\$. Then $$\frac{f_{\mathrm{stop}}}{f_0} = 1.04.$$

Have a look at this fourth order elliptic filter taken from the Wikipedia article.


Although it does not quite meet your requirements you can see it is feasible. A higher order elliptic filter can achieve what you are after.

You should keep in mind that elliptic filters can do disturbing things to the phase of the signal. Since you did not mention anything about your phase constraints, I have assumed that an elliptic filter is suitable.