When should I use normal Q learning over a DQN?
I haven't worked with reinforcement learning but in general one should judge based on the data (or in this case states). The shift of algorithms from normal ones to deep ones is made when we require more power. So it boils down to your need!
If you can get results with simple algorithms always choose that but if simple algorithms don't cut it you will have to go for the deeper ones.
Not going on the true computational meaning but generally deeper models are more complex and one should always avoid complex models where ever possible.
You also mention &quot;slower&quot;. That has a lot of perspective into it so you shouldn't make a final judgement on the models in that aspect!