[0001]    This application is a continuation-in-part (CIP) of copending U.S. patent application Ser. No. 14/642,656 filed. 2015 Mar. 11. 
     
    
       [0002]    Upgrading a hypervisor can involve shutting down the virtual-machines hosted by the hypervisor. Depending on the mission(s) to which the virtual machines have been dedicated, the shutdown may be costly or otherwise unacceptable. To avoid the shutdown, the virtual machines can be migrated to a standby machine, e.g., using a product such as vMotion, available from VMware, Inc. For example, when upgrading the ESX, a hypervisor available from VMware, Inc., the host is put in a maintenance mode that migrates all the virtual machines from the host machine to a standby machine. While the virtual is machines execute on the standby machine, the original host machine can be provided with an updated hypervisor. The virtual machines can be migrated back, completing the upgrade. Of course, if the standby machine has an instance of the updated hypervisor, the return migration may be omitted. 
         [0003]    Relying on migration to a standby machine to avoid shutting down virtual machines can be problematic. First of all, the required standby machine may not be available. Also, if the number of virtual machines is great and/or if their average size is large, each migration may consume considerable network bandwidth for an extended duration, depriving other network nodes of the bandwidth they may need. For example, a large virtual-machine system can include more than 100 gigabytes (GB) that must be migrated. Accordingly, there remains a need for a less burdensome approach to upgrading (or otherwise updating or exchanging) a hypervisor. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0004]      FIG. 1  is a state sequence chart for a same-machine hypervisor exchange including virtual machine consolidation. 
           [0005]      FIG. 2  is a flow chart of a same-machine upgrade process including virtual-machine consolidation and using suspending and resumption of a host virtual machine. 
           [0006]      FIG. 3  is a flow chart of a same-machine upgrade process including virtual-machine consolidation and using inter-partition migration of a host virtual machine. 
           [0007]      FIG. 4  is a schematic diagram of a computer system that can implement the processes of  FIGS. 2 and 3 . 
       
    
    
     DETAILED DESCRIPTION 
       [0008]    In accordance with the present invention, hypervisors are exchanged without removing or shutting down virtual machines. For example, an upgraded. version of a hypervisor can replace a previous version of the hypervisor. To simplify the exchange, the virtual machines are “consolidated” to reduce the number of virtual machines running on the old hypervisor as the exchange begins. 
         [0009]    For example, in the chart of  FIG. 1 , at time T 1 , a computer system  100  includes a number N of “resident” virtual machines, VM 1 , VM 2  . . . VMN running on an “old” hypervisor  102 , which itself executes on a host machine  104 . During consolidation  151 , a host virtual machine VM 0 , shown in  FIG. 1  at time T 2 , is initiated on old hypervisor  102 . The guest operating-system for virtual machine VM 0  is a “guest” hypervisor  106 . Virtual machines VM 1 -VMN are migrated from old hypervisor  102  to guest hypervisor  106 . From the perspective of old hypervisor  102 , there is only one virtual machine (VM 0 ) at time T 2 . 
         [0010]    Thus, during a hypervisor exchange from old hypervisor  102  to new hypervisor  108  there is, in effect, only one virtual machine (VM 0 ) to “worry about”. The importance of this is explained further below in the context of the various ways of effecting exchange  152 . In any event, as a result of exchange  152 , computer system  100  assumes the configuration associated with time T 3  in  FIG. 1  in which new hypervisor  108  has replaced old hypervisor  102 . 
         [0011]    At  153 , virtual machines VM 1 -VMN are “dissociated” in that they are no longer presented to a hypervisor as a single virtual machine. The dissociation is accomplished by migrating the virtual machines from guest hypervisor  106  to new hypervisor  108 . Virtual machine VM 0  is then terminated. The result is shown in  FIG. 1  for time T 4 . Comparing system  100  at time T 1  and time T 4  shows that old hypervisor  102  has been replaced by new hypervisor  108 . In the case that new hypervisor  108  is an upgrade of old hypervisor  102 , a hypervisor upgrade is achieved without shutting down the hosted VMs or migrating them to a different host machine. 
         [0012]    A hypervisor exchange process  200  is flow-charted in  FIG. 2 . At  201 , virtual machines VM 1 -VMN are executing on an old hypervisor, which is executing on a machine. At  202 , a virtual machine is installed with a “guest” hypervisor as its guest operating system. At  203 , virtual machines VM 1  . . . VMN are migrated from the old hypervisor to the guest hypervisor, implementing consolidation  151  of  FIG. 1 . If the guest hypervisor is the same as the new hypervisor, the guest hypervisor can be used to validate that VM 1 -VMN and any partner software will run well together. Alternatively, the guest hypervisor may be the same as the old hypervisor or may be another hypervisor. 
         [0013]    At  204 ,  FIG. 2 , virtual machine VM 0  is suspended, freezing all processes running thereon including processes associated with virtual machines VM 1  . . . VMN. At  205 , the new hypervisor is loaded onto the machine, terminating the old hypervisor; this implements exchange  152  of  FIG. 1 . In the case that the hypervisors are versions of VMware&#39;s ESX, action  205  uses a technique called loadESX to side-load the new hypervisor on the machine. At  206 ,  FIG. 2 , nesting virtual machine VM 0  is resumed so that virtual machines VM 1  . . . VMN are also resumed. 
         [0014]    At  207 , virtual machines VM 1 -VMN are migrated from the guest hypervisor to the new hypervisor, effecting dissociation  153  of  FIG. 1 . At  208 ,  FIG. 2 , virtual machine VM 0  can be terminated, completing the hypervisor exchange. In the event that the new hypervisor is an upgraded version of the old hypervisor, process  200  can be seen as a same-machine rebootless hypervisor upgrade process. 
         [0015]    An alternative hypervisor exchange process  300  is flow-charted in  FIG. 3 . At  301 , virtual machines VM 1 -VMN are executing on an old hypervisor, which is executing on a machine. At  302 , a virtual machine VM 0  is installed with a “guest” hypervisor as its guest operating system. At  303 , virtual machines VM 1 -VMN are migrated from the old hypervisor to the guest hypervisor, implementing consolidation  151  of  FIG. 1 . If the guest hypervisor is the same as the new hypervisor, the guest hypervisor can be used to validate that VM 1 -VMN and any partner software will run well together. Alternatively, the guest hypervisor may be the same as the old hypervisor or may be another hypervisor. 
         [0016]    At  304 ,  FIG. 3 , the machine is partitioned to form partitions P 1  and P 2 , with source partition P 1  hosting VMs VM 1 -VMN. At  305 , the new hypervisor is loaded onto target partition P 2 . Note that the consolidating  303  can occur before or after the partitioning at  304  and even after the installing at  305 . At  306 ,  FIG. 2 , virtual machines VM 1 -VMN are migrated from the guest hypervisor to the new hypervisor, is effecting exchange  152  of  FIG. 1 . The source partition P 1  is destroyed at  307 . 
         [0017]    At  308 ,  FIG. 3  virtual machines VM 1 -VMN are migrated from the guest hypervisor to the new hypervisor, effecting dissociation  153  of  FIG. 1 . At  309 ,  FIG. 3 , virtual machine VM 0  can be terminated, completing the hypervisor exchange. The new hypervisor can be an upgraded version of the old version, so process  300  can be used as a same-machine hypervisor upgrade process. 
         [0018]    In the case that the hypervisors are versions of VMware&#39;s ESX, process  300  uses a technique called loadESX to side-load the new hypervisor on a partition of the machine and to issue a fast migration from the source partition to the target partition. During this migration, if the virtual machines were rot consolidated, an error could leave the computer system in a state that from which there was no practical recovery. However, because of the consolidation, there is only one virtual machine being migrated; therefore, a failed migration can be resolved, by simply destroying the second partition which will revert the system to a known state. 
         [0019]    One giant advantage of virtualization is that a virtual machine can run anywhere and the underneath hardware can change at any time without the virtual machine being aware of it. Thus, one can easily transform a system with N virtual machines to a system with only one virtual machine by simply creating a nested ESX VM and migrating all the other virtual machines onto it. Once the consolidation is complete, a new partition can be created with a fresh ESX. One can then migrate the nested ESX from the old partition to the new one. Lastly, the source partition can be destroyed, and all the nested ESX virtual machines can be migrated to the host ESX. Here is process  300  in algorithmic form, where the hypervisors are versions of ESX.
   def upgradeESX( ):
       nESX=createNestedESX( )   forall vm in host:
           migrate vm into nESX   
           part=createPartition( )   partESX=loadESX(part)   migrate nESX into partESX   destroyOldPartition( )   forall vm in nESX:
           migrate vm into partESX   
           destroy nESX
 
Note that “loadESX is simply referring to the process of launching another ESXi instance on a subset of the hardware.
   
       
 
         [0031]    Computer system  100  is shown in greater detail in  FIG. 4 . At the time represented in  FIG. 4 , machine  104  is divided to define source partition P 1  and target partition P 2 . Old hypervisor  102  is running on source partition P 1 , while new hypervisor  108  is running on target partition P 2 . Virtual machines VM 1 -VMN are executing on guest hypervisor  106 , which is hosted by virtual machine VM 0 . Virtual machine VM 0  is being migrated from source partition P 1  to target partition P 2 , as at action  306  of process  300 , flow charted in  FIG. 3 . 
         [0032]    Machine  102  includes memory  406 , and storage controllers  408  is and  410  for accessing external storage  412 . Collectively, memory  406  and external storage  412  store substantially all the information defining virtual machines VM 0  and VM 1 -VMN. Migrating the virtual machine is effected. by transferring information from source partition P 1  to target partition P 2 . The virtual machine images in :memory and storage are not moved, rather pointers to memory and storage locations of the images are communicated by source partition P 1  to target partition P 2 . 
         [0033]    Memory  406  includes source-partition memory  414 , target partition memory  416 , and shared memory  418 . Partition P 1  informs target partition P 2  of the locations within memory  414  that contain information needed to migrate a virtual machine. The target partition P 2  then claims that memory so that, in effect, the claimed memory exits source-partition memory  414  and becomes part of target-partition memory  416 , even though no memory physically moves with machine  102 . Source partition P 1  can prepare a list of memory pages and ranges freed as virtual machines are migrated from source partition P 1 . The list can be stored in shared memory  418 , which can be accessed by both partitions. Target partition P 2  can read the list and claim the listed memory. In an alternative embodiment, memory contents are physically moved from memory in source partition P 1  to memory in target partition P 2 . 
         [0034]    Machine  102  includes processors (CPUs)  431 ,  432 ,  433 , and  434 , which. are divided among partitions P 1  and P 2  when the partitions are created. Eventually, however, all memory and devices (storage controllers, NICs, etc.) are to be transferred to the target partition P 2 . However, at least one processor, e.g.,  431 , and some memory  414  is required until very near the end to execute code of old hypervisor  104  to complete the transfer. The last processor  431  makes a final list of memory locations, stores it in shared memory  418 , and shuts down. Target partition P 2  reads the list and claims the memory and the last processor. Also, the target partition. P 2  can reinitialize and claim shared memory. The source partition P 1  is terminated and new hypervisor  108  takes control of all of machine  102 . The resident virtual-machines are migrated to the new hypervisor, and the host VM is destroyed to complete the hypervisor upgrade/exchange process. 
         [0035]    Some devices, such as an inter-processor interrupt controller (IPIC)  440  and an input/output memory management unit (IOMMU)  442  may be required by both partitions during VM migration. To avoid conflicts, access to these devices may be controlled, by respective semaphores i.e., locks). Whichever partition “owns” the semaphore, can use the device. The other partition is excluded until the previous owner releases the semaphore. Once the hypervisor update is complete, the semaphores can be dissolved. It should be noted that process  200  can be implemented on computer system  100  without the partitioning. 
         [0036]    When the virtual machines are transferred to the nested ESXi virtual machine, its storage and networking settings remain the same. For networking, a virtual switch on the underlying ESXi host and the io ESXi VM is configured to provide equivalent functionality. For the case of storage, the same storage is mounted into the virtual machine, assuming that the storage is remote like NFS or some other network share. If the storage is local, a small translation layer can be used so that the blocks in the virtual disks of the virtual machines VM 1 -VMN are the same before and after the migration. 
         [0037]    In an alternate arrangement, there can be more than one nested. ESXi virtual machine. (i.e., there can be an m:n mapping of the number of virtual machines to the number of nested ESXi virtual machines created). There may be situations where moving all the virtual machines into one nested ESX virtual machine causes performance issues. In those cases, the resident virtual machines can be distributed among two or more such ESXi virtual machines. This will still drastically reduce the number of virtual machines that are to be dealt with when switching over from the old version to the new version of ESXi on the physical machine. 
         [0038]    Herein, art labelled “prior art”, if any, is admitted prior art; art not labelled “prior art” is not admitted prior art. The illustrated embodiments as well as variations thereupon and modifications thereto are provided for by the present invention, the scope of which is limited by the following claims.
A hypervisor exchange, e.g., an upgrade, can include consolidating resident virtual machines into a single host virtual machine, exchanging an old hypervisor with a new (upgraded) hypervisor, and disassociating the virtual resident virtual machines by migrating them to the new hypervisor. The consolidating can involve migrating the resident virtual machines from the old hypervisor to a guest hypervisor on the host virtual machine. The exchange can involve: 1) suspending the host virtual machine before the exchange; and 2) resuming the host virtual machine after the exchange; or migrating the host virtual machine from a partition including the old hypervisor to a partition hosting the new hypervisor. Either way, an exchange (upgrade) is achieve without requiring a bandwidth consuming migration over a network to a standby machine.