why do rechargeable batteries fail so horribly in high drain environments?
It's hard to generalize to a battery type because any poorly made mass production battery has the risk of catastrophic failure in high drain situations just because of poor quality control. 

Having said that, it is generally my experience that NiCd and NiMh perform quite superiorly in high drain situations compared to alkaline.  This is almost entirely due to their lower internal resistance compared to alkalines. The higher the internal resistance of a battery, the harder it is for the battery to provide higher currents and the hotter it will get when it provides those currents.  When the battery overheats, it can vent and warp and leak and lose capacity.

Many of the hybrid vehicles use NiMh batteries and they are used in a very high drain situation; yet the Prius for example has had very few battery failures in its 14 year history. 

I have been using high quality nimh rechargeable AA and AAA in my high drain flashlights for a couple of years without a single failure. I suspect you have just had bad luck with your rechargeables? Or possibly let them drain too far or over charge them which could lead to the catastrophic failures.