Deep Q-Learning for physical quantity: q-values distribution not as expected
One guess is is that network isn't lucky enough to encounter (sufficient number of times) an example that leads to the reward.  The issue is mentioned here pointing out that RL-agent might favor higher though incorrect estimates too early during learning. Link hypotheses that perhaps "step-annealing"  could be better, instead of gradualy annealing the epsilon-greedy parameter.

In other words, every time you reduce the epsilon-greedy parameter, you reduce the network's chance to take a random action and actually encounter your (already rare by default) reward. Consider reducing the epsion-greedy parameter further in time, or maybe use a step function to anneal it.

There is also Hindsight experience replay. It tells you to concatenate to your input-vector the state in which you wish to ultimately end up (best-case scenario), and feed it into the network, for forward prop. 

After it fails to encounter your reward, you take the state where the network ended up, and backpropagate as if that state was what you actually aimed for originally.  

According to the above link you can also give it part of the state vector, if the exact destination state is unknown.

HER teaches your network to achieve smaller goals instead of "fixating" on the best-possible goal. Unlike the traditional approach "don't care how, just give me reward", HER describes how the final state looks like. Now, the network is able to learn how states relate to each other, and ultimately how they relate to the destination state the network was meant to go. (instead of mere "I failed" or "I succeeded")

Traditionally, your network won't get a chance to form an impression of the  environment if encountered rewards are zero. With HER, you get such a chance even if the end goal is not visible on the horizon and rewards would be zero or uniform.

Here is a great blog about HER

Video of Hindsight experience replay