Definitions of autocatalysis
I can't tell if the modifier "formal" applies only to "definition" or to "algorithm" also, in the first sentence of your question's body. In case any algorithm would do: this looks like a pretty standard instance of the cycle-detection problem that one learns in undergraduate computer science.

If you only want to detect the presence of cycles, then the algorithm is essentially just to attempt a topological sorting of the graph; if the attempt fails, you have a cycle, and I guess that would mean that you have some sort of catalysis going on. (I didn't major in chemistry; please yell at me if that's non sequitur.)

Okay, so mere cycles are not enough - it seems you want cycles that have "gain". How about a tortoise and hare and have these multiply a state variable by the ratio of outgoing to incoming edges to reaction nodes (denoted by squares in your diagram)? If the hare catches up to the tortoise, that implies a cycle, hence catalysis; then if you compare their state, if the hare has accumulated a larger "gain" than the tortoise, I would say that that shows the presence of autocatalysis?

I'm not sure how this works if there are multiple, linked cycles. You could have one cycle in your graph that's autocatalytic, and another cycle linked to it that "uses up" its catalyst (something of a contradiction). For example:

B + C -&gt; D + E
A + D + H -&gt; 2C + F
2F + G -&gt; H


Is that still an autocatalytic system?