Calculate Q parameter for Deep Q-Learning applied to videogames
The neural network you need to implement for Q-learning must approximate the Q function $Q(s,a)$. There are two ways to do this:


Using $(s,a)$ as input. A lot of the literature will assume this for simplicity, and you will see notations like $\hat{q}(s,a,\theta)$ to show you are approximating the function, and have parameters $\theta$ to learn for the neural network.
Using $(s)$ as input, and have the network output multiple $Q(s,a)$ values - one for each possible action. E.g. if you have four possible actions, then your output might be $[\hat{q}_0, \hat{q}_1, \hat{q}_2, \hat{q}_3]$. This can be more efficient, since in order to find the best, maximising, action, you need all the action values calculated.


Once you have a neural network set up like this, and a table of history (that grows on each action actually taken), this is how you make use of it:

For each sampled [state, action, reward, future_state]:
  Calculate td_target = reward + gamma * max_a'(Q(future_state, a')):
    Run NN(future_state, a') forward for each possible action a' and find max
  Train the NN using the inputs (state, action) and desired output td_target


You would use this variant if the network output multiple Q values at once:

For each sampled [state, action, reward, future_state]:
  Calculate td_target = reward + gamma * max_a'(Q(future_state, a')):
    Run the NN(future_state) forward and take the max output
  Construct the desired output:
    Run the NN(state) forward to get array e.g. [q0, q1, q2, q3]
    Substitute td_target depending on action e.g. [q0, q1, td_target, q3]
  Train the NN using the inputs (state) to learn the desired output


Although this second approach looks more complex, it avoids looping (or mini-batching) over multiple a' values to find a max, so can be more efficient.

If you are using a frozen copy of NN to help with stability (a common feature in DQN), then you should use the frozen copy exclusively to calculate TD targets, and the learning copy to calculate current outputs.

Note it is important that you don't store, but instead re-calculate Q values  at all times. That is both because the initial values will be incorrect (depending on how the NN was initialised), and also that they should improve over time as the agent learns a better policy.

Another important detail is that you should not use the NN to calculate Q value for terminal states. If future_state is terminal, then instead of using the NN, treat max_a'(Q(future_state, a')) as zero. Typically just detect that this is a terminal state and hard-code a 0 somehow for it.