anchor.diff:  == Advanced Topics
   
   Blue-green deployments are a proven approach to reducing downtime caused while
  -updating an environment.  This is done by creating a parallel environment on
  -which the new deployment can be installed.  If a problem is detected and after
  +updating an environment. This is done by creating a parallel environment on
  +which the new deployment can be installed. If a problem is detected and after
   the new deployment is verified, traffic can be switched over with the option to
   rollback.
   
   While blue-green is a valid strategy for deploying just about any software,
  -there are always trade offs to be made.  Not all environments have the same
  -uptime requirements or the resources to properly perform blue-green
  -deployments.  In an {product-title} environment the most suitable candidate for
  -blue-green deployments are the Nodes.  Uptime is most important for these
  -workloads and the additional complexity of blue-green deployments can be
  -justified.  Often the main hurdle to overcome is having the excess capacity to
  -facilitate such an approach.
  +there are always trade-offs to be made. Not all environments have the same
  +uptime requirements or the resources to properly perform blue-green deployments.
  +In an {product-title} environment the most suitable candidate for blue-green
  +deployments are the Nodes. Uptime is most important for these workloads and the
  +additional complexity of blue-green deployments can be justified. Often the main
  +challenge is having the excess capacity to facilitate such an approach.
   
   ifdef::openshift-enterprise[]
   Another lesser challenge is that the administrator will have to temporarily
   share the entitlements between the blue-green deployments or provide access to
  -the installation content by means of a system such as Red Hat Satellite.  This
  +the installation content by means of a system such as Red Hat Satellite. This
   can be accomplished by sharing the consumer ID from the previous host.
   
   On the old host:
   ----
  -subscription-manager identity | grep system
  +# subscription-manager identity | grep system
   system identity: 6699375b-06db-48c4-941e-689efd6ce3aa
   ----
   
   On the new host:
   ----
  -subscription-manager register --consumerid=6699375b-06db-48c4-941e-689efd6ce3aa
  +# subscription-manager register --consumerid=6699375b-06db-48c4-941e-689efd6ce3aa
   ----
   
   [IMPORTANT]
   ====
  -When the deployment has been declared successful, it is important that the old
  -host be unregistered with `*subscription-manager clean*` to prevent the
  -environment from being out of compliance.
  +After a successful deployment, remember to unregister the old host with
  +`*subscription-manager clean*` to prevent the environment from being out of
  +compliance.
   ====
   
   endif::openshift-enterprise[]
   
  +[[manual-upgrades-preparing-for-upgrade]]
   === Preparing for Upgrade
   
  -xref:#upgrading-masters[After the master and etcd servers have been upgraded]
  -you will need to ensure that your current production nodes are labeled either
  -blue or green.  In this example the current installation will be blue and the
  -new environment will be green.
  -
  -----
  -oc label --all nodes color=blue
  -----
  -
  -There is quite a bit of flexibility here.  Perhaps there was only some subset of
  -nodes that required the uptime guarantees of a blue-green deployment.  In that
  -case, the `-l` flag can be used to match a subset of the environment using a
  -selector.
  +xref:#upgrading-masters[After the master and etcd servers have been upgraded],
  +you need to ensure that your current production nodes are labeled either blue or
  +green. In this example, the current installation will be blue and the new
  +environment will be green.
  +
  +----
  +$ oc label --all nodes color=blue
  +----
  +
  +In the case of nodes requiring the uptime guarantees of a blue-green deployment,
  +the `-l` flag can be used to match a subset of the environment using a selector.
   
   Next, xref:../install/advanced_install.adoc#adding-nodes-advanced[the green
  -environment] should be created for any nodes that are to be replaced.  The
  -`*color=green*` label can be applied by Ansible using the `*openshift_node_labels*`
  -variable for each node.  The most critical attribute of the node at this step
  -is that the `*openshift_schedulable=false*` variable be set.  The goal is for no
  -workloads to be scheduled until these nodes are determined to be healthy.
  -
  -Once the green nodes are in `*Ready*` state, they can be made scheduleable. The
  -blue nodes are then disabled so that no new pods are run on them.
  -
  -----
  -oadm manage-node --schedulable=true --selector=color=green
  -oadm manage-node --schedulable=false --selector=color=blue
  +environment] should be created for any nodes that are to be replaced. Ansible
  +can apply the `*color=green*` using the `*openshift_node_labels*` variable for
  +each node.
  +
  +In order to delay workload scheduling until the nodes are
  +xref:../dev_guide/application_health.adoc#dev-guide-application-health[healthy],
  +be sure to set the `*openshift_schedulable=false*` variable.
  +
  +Once the green nodes are in *Ready* state, they can be made schedulable. Then,
  +blue nodes are disabled so that no new pods are run on them.
  +
  +----
  +# oadm manage-node --schedulable=true --selector=color=green
  +# oadm manage-node --schedulable=false --selector=color=blue
   ----
   
   A common practice is to scale the registry and router pods until they are
  -migrated to the green nodes.  For these pods, a canary deployment approach is
  -commonly used.  Scaling them up will make them immediately active on the new
  -nodes.  Pointing the deployment configuration to the new image causes a rolling
  -update to happen.  However, because of node anti-affinity, and the fact that the
  +migrated to the green nodes. For these pods, a _canary_ deployment approach is
  +commonly used. Scaling them up will make them immediately active on the new
  +nodes. Pointing the deployment configuration to the new image initiates a
  +rolling update. However, because of node anti-affinity, and the fact that the
   blue nodes are still unschedulable, the deployments to the old nodes will fail.
   At this point, the registry and router deployments can be scaled down to the
  -original number of pods.  At any given point the original number of pods is
  +original number of pods. At any given point, the original number of pods is
   still available so no capacity is lost.
   
  +[[manual-upgrades-warming-the-new-nodes]]
   === Warming the New Nodes
   
   In order for pods to be migrated from the blue environment to the green, the
  -images must be pulled.  Network latency and and load on the registry can cause
  -delays if there is not sufficient capacity built in to the environment. Often
  -the best way to minimize impact to the running system is trigger new pod
  -deployments that will land on the new nodes.  This can be accomplished by
  -importing new ImageStreams.
  -
  -Quite often, a major release of {product-title} is the motivation for a blue-green
  -deployment.  When this occurs, there will be new ImageStreams available for users of
  -Source-to-Image.  Upon import, any builds or deployments configured with
  -ImageChangeTriggers will be automatically created.
  +images must be pulled. Network latency and load on the registry can cause delays
  +if there is not sufficient capacity built in to the environment. Often, the best
  +way to minimize impact to the running system is to trigger new pod deployments that
  +will land on the new nodes. Accomplish this by importing new *ImageStreams*.
  +
  +A major release of {product-title} is the motivation for a
  +blue-green deployment. At that time, new *ImageStreams* become
  +available for users of Source-to-Image. Upon import, any builds or deployments
  +configured with *ImageChangeTriggers* are automatically created.
   
   It is important to realize that this process can trigger a large number of
  -builds.  The good news is that the builds are performed on the green nodes and,
  +builds. The good news is that the builds are performed on the green nodes and,
   therefore, do not impact any traffic on the blue deployment.
   
   Monitor build progress:
   
   ----
  -oc get events -w --all-namespaces
  -----
  -
  -In large environments, builds rarely completely stop.  You should, however, see
  -a large increase and descrease caused by the administrative import.
  +$ oc get events -w --all-namespaces
  +----
  +
  +In large environments, builds rarely completely stop. However, you should see a
  +large increase and decrease caused by the administrative import.
   
   Another benefit of triggering the builds is that it does a fairly good job of
   fetching the majority of the ancillary images to all nodes such as the various
  -build images, the pod infrastructure image, and deployers.  Everything else can be
  -moved over using node evacuation and will proceed more quickly as a result.
  -
  +build images, the pod infrastructure image, and deployers. Everything else can
  +be moved over using node evacuation and will proceed more quickly as a result.
  +
  +[[manual-upgrades-node-evacuation]]
   === Node Evacuation
   
  -For larger deployments, it is quite possible to have other labels that help
  -determine how evacuation can be coordinated.  The most conservative approach
  -for avoiding downtime is to evacuate one node at a time.  If services are
  +For larger deployments, it is possible to have other labels that help
  +determine how evacuation can be coordinated. The most conservative approach
  +for avoiding downtime is to evacuate one node at a time. If services are
   composed of pods using zone anti-affinity, then an entire zone can be
  -evacuated at once.  It is important to ensure that the storage volumes used are
  +evacuated at once. It is important to ensure that the storage volumes used are
   available in the new zone as this detail can vary among cloud providers.
   
  -In OSE 3.2 and later, a node evacuation is triggered whenever the service is
  -stopped.  Manual evacuation and deletion of all blue nodes at once can be
  -achieved as follows:
  -
  -----
  -oadm manage-node --selector=color=blue --evacuate
  -oc delete node --selector=color=blue
  -----
  -
  +In OpenShift Enterprise 3.2 and later, a node evacuation is triggered whenever
  +the service is stopped. Achieve manual evacuation and deletion of all blue nodes
  +at once by:
  +
  +----
  +# oadm manage-node --selector=color=blue --evacuate
  +# oc delete node --selector=color=blue
  +----
  +
  +[[manual-upgrades-summary]]
   === Summary
   
   In the {product-title} environment, the most suitable candidate for blue-green
  -deployments is the node tier.  All user processes run on these systems and even
  -critical pieces of {product-title} infrastructure are self-hosted there.  Uptime is
  -most important for these workloads and the additional complexity of blue-green
  -deployments can be justified.  The exact implementation of this approach
  -varies based on your requirements.
  +deployments is the node tier. All user processes run on these systems and even
  +critical pieces of {product-title} infrastructure are self-hosted there. Uptime
  +is most important for these workloads and the additional complexity of
  +blue-green deployments can be justified. The exact implementation of this
  +approach varies based on your requirements.
  
positive.code: 
  During the upgrade, it can sometimes be useful to take a master out of rotation
  since some DNS client libraries will not properly to the other masters for
  cluster DNS.  In addition to stopping the master and controller services, you
  can remove the EndPoint from the kubernetes service's `*subsets.addresses*`.
  
  ----
  oc edit ep/kubernetes -n default
  ----
  
  When the master is restarted, the kubernetes Service will be automatically
  updated.
  
  [[updating-policy-definitions]]
  
negative.code: 
  After a cluster upgrade, the recommended
  xref:../../architecture/additional_concepts/authorization.adoc#roles[default
  cluster roles] may be updated. To check if an update is recommended for
  your environment, you can run:
  
  ----
  