What happens when microcontrollers run out of RAM?
In general the stack and the heap crash in to each other.  At that point it all gets messy.

Depending on the MCU one of several things may (or will) happen.


Variables get corrupted
The stack gets corrupted
The program gets corrupted


When 1 happens you start getting strange behaviour - things not doing what they should.
When 2 happens all manner of hell breaks loose.  If the return address on the stack (if there is one) is corrupted, then where the current call will return to is anyone's guess. At that time basically the MCU will start doing random things.
When 3 happens again, who knows quite what would happen.  This only happens when you're executing code out of RAM.

In general when the stack gets corrupted it's all over.  Just what happens is down to the MCU.

It might be that trying to allocate the memory in the first place fails so the corruption doesn't happen.  In this case the MCU might raise an exception.  If there is no exception handler installed, then most often the MCU will just halt (an equivalent of while (1);.   If there is a handler installed, then it might reboot cleanly.

If the memory allocation does go ahead, or if it tries, fails, and just continues with no memory allocated, then you're into the realms of "who knows?".  The MCU might end up rebooting itself through the right combination of events (interrupts caused that end up resetting the chip, etc), but there's no guarantee of that happening.

What there can usually be a high probability of happening, though, if it's enabled, is the internal watchdog timer (if one exists) timing out and rebooting the chip.  When the program goes completely AWOL through this kind of crash the instructions to reset the timer generally won't be run, so it will time out and reset.