anchor.diff:  
   After a cluster upgrade, the recommended
   xref:../../architecture/additional_concepts/authorization.adoc#roles[default
  -cluster roles] may have been updated. To check if an update is recommended for
  +cluster roles] may be updated. To check if an update is recommended for
   your environment, you can run:
   
   ----
  
positive.code: == Advanced Topics
  
  Blue-green deployments are a proven approach to reducing downtime caused while
  updating an environment.  This is done by creating a parallel environment on
  which the new deployment can be installed.  If a problem is detected and after
  the new deployment is verified, traffic can be switched over with the option to
  rollback.
  
  While blue-green is a valid strategy for deploying just about any software,
  there are always trade offs to be made.  Not all environments have the same
  uptime requirements or the resources to properly perform blue-green
  deployments.  In an {product-title} environment the most suitable candidate for
  blue-green deployments are the Nodes.  Uptime is most important for these
  workloads and the additional complexity of blue-green deployments can be
  justified.  Often the main hurdle to overcome is having the excess capacity to
  facilitate such an approach.
  
  ifdef::openshift-enterprise[]
  Another lesser challenge is that the administrator will have to temporarily
  share the entitlements between the blue-green deployments or provide access to
  the installation content by means of a system such as Red Hat Satellite.  This
  can be accomplished by sharing the consumer ID from the previous host.
  
  On the old host:
  ----
  subscription-manager identity | grep system
  system identity: 6699375b-06db-48c4-941e-689efd6ce3aa
  ----
  
  On the new host:
  ----
  subscription-manager register --consumerid=6699375b-06db-48c4-941e-689efd6ce3aa
  ----
  
  [IMPORTANT]
  ====
  When the deployment has been declared successful, it is important that the old
  host be unregistered with `*subscription-manager clean*` to prevent the
  environment from being out of compliance.
  ====
  
  endif::openshift-enterprise[]
  
  === Preparing for Upgrade
  
  xref:#upgrading-masters[After the master and etcd servers have been upgraded]
  you will need to ensure that your current production nodes are labeled either
  blue or green.  In this example the current installation will be blue and the
  new environment will be green.
  
  ----
  oc label --all nodes color=blue
  ----
  
  There is quite a bit of flexibility here.  Perhaps there was only some subset of
  nodes that required the uptime guarantees of a blue-green deployment.  In that
  case, the `-l` flag can be used to match a subset of the environment using a
  selector.
  
  Next, xref:../install/advanced_install.adoc#adding-nodes-advanced[the green
  environment] should be created for any nodes that are to be replaced.  The
  `*color=green*` label can be applied by Ansible using the `*openshift_node_labels*`
  variable for each node.  The most critical attribute of the node at this step
  is that the `*openshift_schedulable=false*` variable be set.  The goal is for no
  workloads to be scheduled until these nodes are determined to be healthy.
  
  Once the green nodes are in `*Ready*` state, they can be made scheduleable. The
  blue nodes are then disabled so that no new pods are run on them.
  
  ----
  oadm manage-node --schedulable=true --selector=color=green
  oadm manage-node --schedulable=false --selector=color=blue
  ----
  
  A common practice is to scale the registry and router pods until they are
  migrated to the green nodes.  For these pods, a canary deployment approach is
  commonly used.  Scaling them up will make them immediately active on the new
  nodes.  Pointing the deployment configuration to the new image causes a rolling
  update to happen.  However, because of node anti-affinity, and the fact that the
  blue nodes are still unschedulable, the deployments to the old nodes will fail.
  At this point, the registry and router deployments can be scaled down to the
  original number of pods.  At any given point the original number of pods is
  still available so no capacity is lost.
  
  === Warming the New Nodes
  
  In order for pods to be migrated from the blue environment to the green, the
  images must be pulled.  Network latency and and load on the registry can cause
  delays if there is not sufficient capacity built in to the environment. Often
  the best way to minimize impact to the running system is trigger new pod
  deployments that will land on the new nodes.  This can be accomplished by
  importing new ImageStreams.
  
  Quite often, a major release of {product-title} is the motivation for a blue-green
  deployment.  When this occurs, there will be new ImageStreams available for users of
  Source-to-Image.  Upon import, any builds or deployments configured with
  ImageChangeTriggers will be automatically created.
  
  It is important to realize that this process can trigger a large number of
  builds.  The good news is that the builds are performed on the green nodes and,
  therefore, do not impact any traffic on the blue deployment.
  
  Monitor build progress:
  
  ----
  oc get events -w --all-namespaces
  ----
  
  In large environments, builds rarely completely stop.  You should, however, see
  a large increase and descrease caused by the administrative import.
  
  Another benefit of triggering the builds is that it does a fairly good job of
  fetching the majority of the ancillary images to all nodes such as the various
  build images, the pod infrastructure image, and deployers.  Everything else can be
  moved over using node evacuation and will proceed more quickly as a result.
  
  === Node Evacuation
  
  For larger deployments, it is quite possible to have other labels that help
  determine how evacuation can be coordinated.  The most conservative approach
  for avoiding downtime is to evacuate one node at a time.  If services are
  composed of pods using zone anti-affinity, then an entire zone can be
  evacuated at once.  It is important to ensure that the storage volumes used are
  available in the new zone as this detail can vary among cloud providers.
  
  In OSE 3.2 and later, a node evacuation is triggered whenever the service is
  stopped.  Manual evacuation and deletion of all blue nodes at once can be
  achieved as follows:
  
  ----
  oadm manage-node --selector=color=blue --evacuate
  oc delete node --selector=color=blue
  ----
  
  === Summary
  
  In the {product-title} environment, the most suitable candidate for blue-green
  deployments is the node tier.  All user processes run on these systems and even
  critical pieces of {product-title} infrastructure are self-hosted there.  Uptime is
  most important for these workloads and the additional complexity of blue-green
  deployments can be justified.  The exact implementation of this approach
  varies based on your requirements.
  
negative.code: 
  During the upgrade, it can sometimes be useful to take a master out of rotation
  since some DNS client libraries will not properly to the other masters for
  cluster DNS. In addition to stopping the master and controller services, you
  can remove the EndPoint from the Kubernetes service's `*subsets.addresses*`.
  
  ----
  $ oc edit ep/kubernetes -n default
  ----
  
  When the master is restarted, the Kubernetes service will be automatically
  updated.
  
  [[updating-policy-definitions]]
  