Why is my implementation of Q-learning not converging to the right values in the FrozenLake environment?
I was able to solve the problem.
The main issue for non-convergence was that I was not decaying the learning rate appropriately. I put a decay rate of $-0.00005$ on the learning rate lr, and subsequently Q-Learning also converged to the same value as value iteration.