Change in Training Accuracy
The problem with your second approach (Case 2) is that the neural network might decide to output very large numbers for keypoints near or on the edge.  If it does, the loss function won't do anything to discourage it from doing that.

For instance, suppose the keypoint is at (30,94) for one instance in the training set.  Then outputting (30,1000) has the same penalty as (30,96).  So a network might just learn to output (30,96), and then start increasing the second coordinate (it learns (30,97), then (30,98), etc.) -- there's no penalty for that.

In particular, suppose that at some point the network is outputting (30,96) on that instance, and suppose the gradients encourage increasing the 96 a little bit.  Then in the next iterations the weights will be adjusted so it outputs (30,97).  Because the output is clamped before applying the loss function, (30,97) will receive the same penalty as (30,96).  Also it will have the same gradient.  Thus the learning process will continue adjusting the weights to cause the second coordinate to keep increasing forever.

This causes learning to fail to converge, or to converge very slowly.  And if learning fails to converge, it's plausible that overall rms error (including on other instances where this issue doesn't arise) might be very bad.