Improving DQN with fluctuations
It is not clear form your question, how you use your replay buffer. Basically, you have to store all states/reward tuples and train your agent on the entire buffer.
Moreover, you should give the agent time to explore (all) states of the world. But if you want to speed up training, you can try to implement importance sampling