TF2.0 DQN has a loss of 0?
OK, a bit more digging led me to this: 

The TD target or "target value" gets its name because by updating a Q table or training a NN with it as a ground truth, the estimator will output values in future closer to the supplied value. The estimator "gets closer to the target".

So, I have to update the prediction network with the "ground truth".

    for i, val in enumerate(actions):
        predict_q[i][val] = target_q[i]