How does Q-Learning deal with mixed strategies?
One possibility is to use softmax and choose each action a randomly with probabiliy $p = \frac{\exp(Q(s,a))}{\sum_a \exp(Q(s,a))}$. I don't thinks it is still Q-learning though.