Magic: The Gathering Sorcery vs Activated Abilities
There is one stack, that all spells and abilities use (with the exception of special actions and mana abilities). In your specific example, player A casts Whelming Wave, and once they finish casting it, it's on the stack. Then, player B activates their creature's ability "in response" to the Whelming Wave, meaning "while the Whelming Wave is on the stack". That ability goes on the stack on top of the Whelming Wave. Then the ability resolves, and the ability does whatever it does. Finally, the Whelming Wave resolves, and, the creatures are returned to their owners' hands. If, however, the resolving ability killed player A, the Whelming Wave would immediately leave the game with its controller, and it would never resolve.

This can happen because abilities can be activated at instant speed; in this case, they can be activated in response to other spells or abilities. Rule 112.3b says


  Activated abilities have a cost and an effect. They are written as “[Cost]: [Effect.] [Activation instructions (if any).]” A player may activate such an ability whenever he or she has priority. Doing so puts it on the stack, where it remains until it’s countered, it resolves, or it otherwise leaves the stack. See rule 602, “Activating Activated Abilities.”


Notable exceptions are the Equip and Outlast abilities, which can only be activated at sorcery speed.