How to cancel an order transition if failure is detected in Commerce 2 / State Machine
We don't have a "cancel" or "reverse" feature right now, but it would be worth a feature request in the State Machine queue. The main thing you / we'll need to overcome is what to do in the event a transition in progress has already saved data, triggered 3rd party services, etc. Presumably, there are certain things that should prevent a transition from being reversed (e.g. I've refunded a payment card and therefore must move the order to a canceled or at least unpaid state). However, other things may fall somewhere in the middle ... be allowable and not prevent reversal.

Lisa had a great idea, too, in that you could define the reverse transition explicitly yourself and trigger it in the event that you had a failure condition in your active transition. That would allow you to encode your own logic for checking / reversing transition side effects.