Using the opponent's mixed strategy in estimating the state value in minimax Q learning
My understanding is now that the author's formula is deliberate. It seeks to learn a worst-case maximizing policy. The formula I instead suggest would, I believe, instead be Nash Q learning where the agent seeks to learn to play a Nash equilibrium.

After debugging, I have gotten good results with the second formula but cannot speak for the original Minimax Q learning one.