How does the DQN loss from td_targets against q_values make sense? Why td_loss is calculated from (td_targets against q_values)?

Why I am lost is because:


q_values is just the probability of action. It does not have a reward and discount.
td_targets does have rewards + discounts * next_q_values. Somemore next_q_values is next state.


How both td_targets and q_values can minus (or Huber or MSE) to get lost work?

td_error = valid_mask * (td_targets - q_values)
td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)


td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)
First of all, DQN is off-policy learning. That means, you are following the behavior policy(epsilon greedy policy) but still learning about the optimal policy or target policy(greedy policy). Td_target in DQN is the estimation of our current state's optimal action-value function independent of the policy we are following (since we are picking next state's action-value from target policy) and q_values(as you referred) is what you get following the behavior policy. While using this kind of update, you are improving both the behavior policy and the target policy.