What's the reason that the alarm clock isn't working when the phone is turned off? I know that the alarm isn't working when the phone is turned off, but my question is: Why doesn't this work? What were the developers/engineers thinking?

I always used my mobile phones as alarm clocks and even very old mobile phone support alarm clocks when they are turned off.
There is no power to the phone when it's off.  It's actually off.  Some phones have a minimalistic charging mode that just shows a battery animation, but I have yet to see an Android device that boots into a clock-only type mode.

The Android OS is designed to use minimal power when not in use.  If you have a problem with the battery dying overnight, it's probably due to a third-party app.