What stops an assembly program from crashing the operating system?
In the end, all programs are machine code, regardless of whether the source language was assembler or a high-level language.

The important thing is that there are hardware mechanisms that limit what a given process can do, including "messing with" registers that could affect other programs or the operating system itself.

This started with a simple distinction between "user" and "supervisor" modes of operation, and has since evolved into a security architecture with multiple "rings" of privilege.



Here's a very generic example to make it a little more concrete:


In "user mode", a process cannot access memory that hasn't been assigned to its process ID. Memory assigned to other processes and the operating system itself is blocked. This includes the values of registers used by those other processes. This is enforced by the MMU hardware.
Therefore, in "user mode", a process cannot access the MMU control registers.
And obviously, in "user mode", a process cannot change the mode to "supervisor mode" except through a very well-defined mechanism that involves calling an operating system function.


The operating system gets control if the process tries to break any of these rules. Once that happens, the OS can simply halt the offending process, never executing any more of its instructions.