Deep Q-learning, how to set q-value of non-selected actions?
Consider these two depictions of a simple Deep Q Learning Algorithm:


Look at the gradient update steps in both images: we are only taking the gradient with respect to the Q value of the one action, $a_i$. The network outputs $N$ nodes, but since we only care about one of them, you can probably see that all of the computations still work out, for the ground truths of the other output nodes are never used in this gradient update step.
I hope this helps.
Edit: looking at you first solution you listed, this is correct, but i feel you aren’t understanding the actual computation of the gradient. If you work it out, the ground truth values of the other nodes aren’t actually used because the loss function is only written in terms of the $i$th node.