and evolve, despite VM's second class citizen status at IBM. Architecture The CP/CMS architecture was revolutionary for its time. The system consisted of a virtualizing control program (CP) which created multiple independent virtual machines (VMs). Platform virtualization was possible because of two elements of the IBM System/360-67: Segregation of privileged "supervisor state" instructions from normal "problem state" instructions Address translation hardware When a program was running in "problem state," using a privileged instruction or an invalid memory address would cause the hardware to raise an exception condition. By trapping these conditions, CP could simulate the appropriate behavior, e.g. performing I/O or paging operations. A guest operating system, which would run in "supervisor state" on a bare machine, was run in "problem state" under CP. The result was a fully virtualized environment. Each virtual machine had its own set of virtual devices, mapped from the system's real hardware environment. Thus a given dial-up teletype was presented to its VM instance as its virtual console. Note that, in CP-67, certain model-dependent and diagnostic instructions were not virtualized, notably the DIAG instruction. Ultimately, in later development at IBM and elsewhere, DIAG instructions were used to create a non-virtualized interface, to what became called a hypervisor. Client operating systems could use this mechanism to communicate directly with the control program; this offered dramatic performance improvements. Any S/360 operating system could in fact be run under CP, but normal users ran Cambridge Monitor System (CMS), a simple, single-user operating system. CMS allowed users to run programs and manage their virtual devices. CP-67 versions 1 and 2 did not support virtual memory inside a virtual machine. This was added in version 3. At that point, testing and development of CP itself could be done by running a full copy of CP/CMS inside a single virtual machine. Some CP/CMS operating system work, such as CP-370 development and MVS testing, ran four- or five-level deep stacks of hardware and OS simulations. The CP/CMS design is different from IBM's previous monolithic operating systems, it separates complex "big system" (dispatching, hardware management, mass storage) from "little system" (application program execution, file I/O, console input/output). The re-categorization of both systems into their own entities prevents a bug in one users' system from affecting both. This is a model feature in microkernel operating systems. IBM's decision to implement virtualization and virtual memory features in the subsequent S/370 design (although missing from the