What's the best reference for rigorous study of mixed-strategy nash equilibrium?
The book by Gibbons that Tobias mentions is titled "A Primer in Game Theory" in Europe. It is excellent, especially in its coverage of more advanced games of imperfect/incomplete information. But it is quite concise in its treatment of games of complete information like the ones you are studying. The book by Osborne might therefore be a better choice. Both of these books are aimed at the introductory graduate level and are therefore quite rigorous.*

However, I feel compelled to say that it sounds like you are taking a first course in game theory, so I would be inclined to suggest also the book "Strategies and Games: Theory and Practice" bu Dutta. It is more verbose, but I think also provides a more gentle introduction to topics like Nash equilibrium and mixed strategies. Game theory, like much of economics is often best learnt by first obtaining a firm grasp of the intuition, and later supplementing it with sufficient rigor.



(*) There is also "Game Thoery" by Fudenberg and Tirole, which is less of a textbook and more of a reference. But it is, perhaps, even more rigorous as a result.