Asynchronous System Trap

Asynchronous System Trap (AST) refers to a mechanism used in several computer operating systems designed by the former Digital Equipment Corporation (DEC) of Maynard, Massachusetts.
Mechanism.
Various events within these systems can be optionally signalled back to the user processes via the AST mechanism. These ASTs act like subroutine calls but they are delivered asynchronously, that is, without any regard to the context of the main thread. Because of this, care must be taken:
ASTs are most commonly encountered as a result of issuing QIO calls to the kernel. Completion of the I/O can be signalled by the issuance of an AST to the calling process/task. Certain runtime errors could also be signalled using the AST mechanism. Within OpenVMS, Special Kernel-Mode ASTs are used as the standard mechanism for getting relatively convenient access to a process context (including getting the process paged into physical memory as may be needed). These types of ASTs are executed at the highest possible per-process priority the next time the scheduler makes that process current, and are used among other things for retrieving process-level information (in response to a $GETJPI "getjob/process information" system call) and for performing process deletion.
The following operating systems implement ASTs:
ASTs are roughly analogous to Unix signals. The important differences are:
VAX/VMS V4 and later implemented an interesting optimization to the problem of synchronizing between AST-level and non-AST-level code. A system service named $SETAST could be used to disable or enable the delivery of ASTs for the current and all less-privileged "access modes" (the OpenVMS term for ring-based security features). However, if the critical section needing protection from ASTs was only a few instructions long, then the overhead of making the $SETAST calls could far outweigh the time to execute those instructions.
So for user mode only (the least privileged ring, normally used by ordinary user programs), a pair of bit flags was provided at a predefined user-writable memory location (in per-process "P1" space). The meanings of these two flags could be construed as "don't deliver any ASTs" and "ASTs have been disabled". Instead of the usual pair of $SETAST calls, the user-mode code would set the first flag before executing the sequence of instructions during which ASTs need to be blocked, and clear it after the sequence. "Then" (note the ordering here, to avoid race conditions) it would check the second flag to see if it had become set during this time: if so, then ASTs really have become disabled, and $SETAST should be called to re-enable them. In the most common case, no ASTs would have become pending during this time, so there would be no need to call $SETAST at all.
The kernel AST delivery code, for its part, would check the first flag before trying to deliver a user-mode AST; if it was set, then it would directly set the ASTs-disabled bit in the process control block (the same bit that would be set by an explicit $SETAST call from user mode), and also set the second flag, before returning and leaving the AST undelivered.
The asynchronous procedure call mechanism in the Windows NT family of operating systems is a similar mechanism.