How to use a loss functions closed-form gradient even when it does not have a closed-form itself
TensorFlow allows for custom gradient functions with tf.custom_gradient.

You could write a decorator that would return specific gradient values for specific values of x. The loss function would not need to be evaluated.