What is the name for this ... (alerting when threshold is exceeded, but only if dropped enough below threshold first)
Hysteresis is a general concept in control systems that seems relevant.

Debouncing is another specific technique used for post-processing data from switches/buttons that has a similar flavor.  The term has sometimes been used elsewhere in software systems to deal with similar issues.

Finally, take a look at rate limiting for a related technique.