Is using passive anti-aliasing filter with its transfer function instead of an active filter common or reliable?
What if instead I make a simple RC filter and set the stop-band again
  to -60dB at 10kHz. Obviously in this case the 0.1dB point will shift
  to a much smaller than 5kHz.


20 dB/decade attenuation from a simple RC will mean it requires a 3 dB point that is 3 decades below 10 kHz i.e. 100 Hz. It's not unheard of to do this and I've done it myself (when using a decent ADC with sufficient resolution that the signal didn't becomes too small and grainy as frequency rose).

However, I would consider going for a 2nd order filter and halving the problem. And, given that it's really easy to add an RC to the output of a 2nd order filter (to make a 3rd order filter), you would improve the cut-off frequency from 100 Hz to 1 kHz.

Alternatively just sample at a significantly higher rate.