How does the DQN loss from td_targets against q_values make sense?
First of all, DQN is off-policy learning. That means, you are following the behavior policy(epsilon greedy policy) but still learning about the optimal policy or target policy(greedy policy). Td_target in DQN is the estimation of our current state's optimal action-value function independent of the policy we are following (since we are picking next state's action-value from target policy) and q_values(as you referred) is what you get following the behavior policy. While using this kind of update, you are improving both the behavior policy and the target policy.