Tensorflow optimizer with negative feedback?
You may do a forward pass, check the loss, and then do backward if you think the loss is acceptable. In TF 1.x it requires some tf.cond and manual calculation and application of gradients. The same in TF 2.0 only the control flow is easier, but you have to use gradient_tape and still apply gradients manually.
I am optimizing a tensorflow model.  It is not a neural net, I am just using tensorflow for easy derivative computations.  In any case, it seems that loss surface has a steep edge somewhere, and my loss will sometimes "pop out" of the local minimum it is currently targeting, the loss will go up a great deal, and the optimizer will go gallivanting off after some other optimum elsewhere.  
I want it to not do that thing.  Specifically, I want it to look at the loss, be all like "holy crap that just went up a whole bunch, I'd better backtrack a bit."  Even though the current gradient may want to send it off elsewhere, I want it to "go back" in sense, and continue trying to find the optimum it was previously targeting.  Is there a tensorflow optimizer that has some kind of "negative feedback" in this way?