Dueling Network gradient with respect to Advantage stream
I verified the following formulas with Gradient Checking, and it's confirming they are correct. (Getting discrepancy less than 0.00002)

Let's draw the tree diagram of what's affecting what:



If we try to find gradient affecting $a_0$, we have to sum-up the pink chains. They make it evident that $a_0$ has affected all $Q$ variables, thus we sum these chains.

So the formula for entire gradient that has to enter $A$ layer is:

$$ 
gradForA =  \left[  \begin{array}{c}
 \frac{\partial E}{\partial a_0}
\\\frac{\partial E}{\partial a_1}
\\\frac{\partial E}{\partial a_2}
\\\frac{\partial E}{\partial a_3}
\end{array}
\right]  = \left[  \begin{array}{c}
 \\\frac{\partial E}{\partial Q_0}(1 - u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(1-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(1-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(1-u)
\end{array}
\right]
$$

where $u$ is $\frac{1}{N}$



Bonus: 

By looking at the above image, we see that the $v_0$ has similarly affected all $Q$ variables, so its gradient is:
$$
gradForV = \frac{\partial E}{\partial Q_0} + \frac{\partial E}{\partial Q_1}+ \frac{\partial E}{\partial Q_2} + \frac{\partial E}{\partial Q_3}
$$

$gradForV$ is a scalar (not a vector) because value stream only has 1 neuron, always.



Bonus #2 

to compute $gradForA$ it's obvious to do via 2 for-loops, nested in one another. However, you can get rid of nesting and improve code performance by re-writing the formula as follows:

$$ 
gradForA =  R + scalarB  
$$

where 

$scalarB = (-u)(\frac{\partial E}{\partial Q_0} + \frac{\partial E}{\partial Q_1}+ \frac{\partial E}{\partial Q_2} + \frac{\partial E}{\partial Q_3})$

and 

$$ R =  \left[  \begin{array}{c}
 \frac{\partial E}{\partial Q_0} 
\\\frac{\partial E}{\partial Q_1} 
\\\frac{\partial E}{\partial Q_2} 
\\\frac{\partial E}{\partial Q_3}
\end{array}
\right]
$$

The cool thing is that $scalarB$ can be computed once (it's a scalar), and re-used when computing the $gradForA$ vector.

Notice that $scalarB$ and $gradForV$ are almost the same thing, just with extra $(-u)$  So you can re-use this value in your code as well.