DQN - how is it possible to train separate outputs for each action? I'm trying to implement a Deep Q Network, but I'm stuck on how you train a network to predict multiple action-values when you can only collect data on a single action.
In the paper it recommends using a different output for each action

We instead use an
architecture in which there is a separate output unit for each possible action, and
only the state representation is an input to the neural network. The outputs correspond
to the predicted Q-values of the individual actions for the input state.

Since we can only visit one action, we only know the loss for that action (ie. a single output). But as far as I'm aware, we need to have values for all of the outputs in order to train the network. What black magic can you use to get the other output values?
It seems like a bad idea to get the network to predict the other action-values and feed them back, as it would affect the optimizer. And if you tried to ignore the other outputs and train it as if there were only the one you were currently focused on, you would still affecting the others as they would share edges.

DQN Paper
The output of your network should be a Q value for every action in your action space (or at least available at the current state). Then you can use softmax or epsilon-greedy (or other strategies) to select the final action. The network will learn to predict which action should return the maximum reward from your current state. Also we update the network after we have collected a specific amount of experience and we use batches from that experience buffer in order to update the network. We do not feed back any value in the vanilla DQN (no recurrence).

There are many good implementations online that can help you understand the algorithm.