Alternative to BIBO stability of a system
For linear systems, BIBO stability is the most useful and practical criterion. For systems described by rational transfer functions it coincides with the condition that all transfer function poles must be located in the left half of the $s$-plane (for continuous-time systems), or inside the unit circle of the $z$-plane (for discrete-time systems). For linear systems, I do not know of any other stability criterion that makes any sense and does not coincide with the BIBO-stability criterion. Since most basic DSP texts focus on linear (and especially time-invariant) systems, you will only find the notion of BIBO-stability there.

Things look a bit different for non-linear systems. There you have several reasonable stability criteria that do not all lead to the same basic criterion as is the case for linear systems. One important notion for non-linear systems is input-to-state stability, which basically means that for zero input, the system is stable about its zero state, and that well-behaved and bounded input signals produce a bounded state trajectory. This article reviews some of these concepts.

But if you're mainly interested in linear systems, BIBO-stability is all you need.