Adversarial Game playing using RL
I would say a good way to make a good agent would be making it play against itself. As you go through several episodes, with a good exploration and exploitation balance, both will gradually learn and converge to Q*(s,a).´


  So will this hold true if the same RL algo, with its learned values
  are used to play some other lesser perfect players?


As long as the states that are played  (or approximations if you are using Function approximation methods) were simulated enough times during  training, it will play well against any kind of opponent. 

If you are training against a completely perfect opponent and you are not using function approximation, I believe you could get to an incomplete Q(s,a) table and as such not be able to predict the best play when facing certain states.