1. A multi-user host computer system for supporting multiple active remote terminals each having their own physical input controls, comprising: a hardware graphics and display subsystem (GPU) having display memory which can store display frames or subframes for multiple active remote terminals and which is accessible to the GPU without the need to perform a GPU context switch; one or more drawing engines for generating display frames which may each correspond to a display frame at one of said remote terminals; tracking means for tracking modified display subframes and for performing selective updates of subframes from said display memory based on this tracking; and means for isolating virtual machines from each other; a host CPU for running more than one CPU virtual machine each of which corresponds to a virtual desktop for a different user, includes a virtual graphics machine having virtual GPU status registers, virtual controls and virtual display memory, and asynchronously generates virtual GPU operations for execution by the physical GPU hardware as if the virtual machine had an exclusive machine view of the hardware GPU; for running a Virtual Machine Monitor which tracks context and state information for each of said virtual graphics machines and their asynchronous operations and achieves proper ordering and interleaving of said virtual GPU operations to the physical GPU hardware; and for controlling access by said remote terminals to said CPU virtual machines; means for connecting said graphics and display subsystem to said host CPU; means for connecting said multiple active remote terminals via a network to said host CPU; means for more than one of said virtual machines to asynchronously access said hardware graphics and display subsystem whereby address translation is performed between the CPU virtual machines and GPU display memory, between the GPU subsystem and the GPU display memory, and between the CPU virtual machines and GPU control registers; and means for managing the selective updates from said GPU display memory by the host CPU so that only the selective updates of said subframes will be transferred via a network subsystem to corresponding ones of said remote terminals.