Should Q values be changing within an epoch/episode or should they change after one episode/epoch? I am trying to use Deep-Q learning environment to learn Super Mario Bros. The implementation is on Github.

I have a neural network that Q values update within an episode for a very small learning rate (0.00005). However, even if I increase the learning rate to 0.00025, the Q values do not change within an episode as they are predicting the same Q values regardless of what state it is in. For example, if Mario moves right, the Q value is the same. When I start a new episode, the Q values change though.

I think that the Q values should be changing within an episode as the game should be seeing different parts and taking different actions. Why don't I observe this?
I was attaining negative values out of my convolutional layers and then using relu on them resulted in the gradient of the activation being 0. Hence, my Q values were not being updated. I've since updated my activations to be ELU. Thanks for the help.