Compute utility of a notification
Good ideas there. But start with simple things first and then ramp up to a more complex system once you have something already working and some human feedback on the solution.

Simpler solution is just a velocity check - if there is X number of failures per Y number of seconds, then send the notification. From your post it seems like 3 failures over 20 second time window should be a good start. It takes a fair bit of tinkering to get this kind of system running - work out the windowing functions, etc.

Good luck :)