What is the name for this ... (alerting when threshold is exceeded, but only if dropped enough below threshold first) Imagine: 
you are monitoring a system, and want an alert when certain measured value goes over a threshold value X.

But perhaps the measured value is constantly fluctuating close to X, and you get a lot of alerts.

There's an solution to that, where you only get new alerts when the measured value has dropped to below Y (e.g. X-10%). So that you only get alerted when the value had dropped meaningfully below X. 

What is the name for that?

I remember this from CS uni years, but can't remember which class it came with (queueing theory?), or what the name for it was. 

I've googled with various keywords (monitoring, maximum, alerts, skewed, ...) but no useful results.
Hysteresis is a general concept in control systems that seems relevant.

Debouncing is another specific technique used for post-processing data from switches/buttons that has a similar flavor.  The term has sometimes been used elsewhere in software systems to deal with similar issues.

Finally, take a look at rate limiting for a related technique.