Confusion about positive feedback condition in gain equation
I'm sure you'll agree intuitively that positive feedback will increase the gain thus |1 + AB| must be &lt; 1 so that |A/(1+AB)| > |A|. Similarly, we can argue that negative feedback will decrease the gain thus requiring |1 + AB| > 1.

The derivations of these formulas assume the systems are Linear (https://en.wikipedia.org/wiki/Linear_time-invariant_theory).The examples you gave that seem to violate this, are cases where the loop gain is so large linearity is no longer satisfied. For example with B = -10 and A = 10000 you find that the closed loop gain is negative, this doesn't make much sense if the open loop gain is positive and if we consider values of AB between 0 and -1 we find the closer to -1 we are the larger the gain gets, yet it is still positive. Then at -1 the gain goes to infinity. Past this point, we can no longer apply those formulas, hence the contradiction.

For more information see, Positive feedback and unstability