Chapter 10. Scaling overcloud nodes
Chapter 10. Scaling overcloud nodes If you want to add or remove nodes after the creation of the overcloud, you must update the overcloud. Note Ensure that your bare metal nodes are not in maintenance mode before you begin scaling out or removing an overcloud node. Use the following table to determine support for scaling each node type: Table 10.1. Scale support for each node type Node type Scale up? Scale down? Notes Controller N N You can replace Controller nodes using the procedures in Chapter 11, Replacing Controller nodes . Compute Y Y Ceph Storage nodes Y N You must have at least 1 Ceph Storage node from the initial overcloud creation. Object Storage nodes Y Y Important Ensure that you have at least 10 GB free space before you scale the overcloud. This free space accommodates image conversion and caching during the node provisioning process. The process for scaling pre-provisioned nodes is similar to the standard scaling procedures. However, the process to add new pre-provisioned nodes differs because pre-provisioned nodes do not use the standard registration and management process from the Bare Metal Provisioning service (ironic) and the Compute service (nova). 10.1. Adding nodes to the overcloud You can add more nodes to your overcloud. Note A fresh installation of Red Hat OpenStack Platform (RHOSP) does not include certain updates, such as security errata and bug fixes. As a result, if you are scaling up a connected environment that uses the Red Hat Customer Portal or Red Hat Satellite Server, RPM updates are not applied to new nodes. To apply the latest updates to the overcloud nodes, you must do one of the following: Complete an overcloud update of the nodes after the scale-out operation. Use the virt-customize tool to modify the packages to the base overcloud image before the scale-out operation. For more information, see the Red Hat Knowledgebase solution Modifying the Red Hat Linux OpenStack Platform Overcloud Image with virt-customize . Procedure Create a new JSON file called newnodes.json that contains details of the new node that you want to register: Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Register the new nodes: Launch the introspection process for each new node: Use the --provide option to reset all the specified nodes to an available state after introspection. Replace <node_1> , <node_2> , and all nodes up to <node_n> with the UUID of each node that you want to introspect. Configure the image properties for each new node: 10.2. Scaling up bare-metal nodes To increase the count of bare-metal nodes in an existing overcloud, increment the node count in the overcloud-baremetal-deploy.yaml file and redeploy the overcloud. Prerequisites The new bare-metal nodes are registered, introspected, and available for provisioning and deployment. For more information, see Registering nodes for the overcloud and Creating an inventory of the bare-metal node hardware . Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Open the overcloud-baremetal-deploy.yaml node definition file that you use to provision your bare-metal nodes. Increment the count parameter for the roles that you want to scale up. For example, the following configuration increases the Object Storage node count to 4: Optional: Configure predictive node placement for the new nodes. For example, use the following configuration to provision a new Object Storage node on node03 : Optional: Define any other attributes that you want to assign to your new nodes. For more information about the properties you can use to configure node attributes in your node definition file, see Bare-metal node provisioning attributes . If you use the Object Storage service (swift) and the whole disk overcloud image, overcloud-hardened-uefi-full , configure the size of the /srv partition based on the size of your disk and your storage requirements for /var and /srv . For more information, see Configuring whole disk partitions for the Object Storage service . Provision the overcloud nodes: Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud . Include the --network-config argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml . Note If you upgraded from Red Hat OpenStack Platform 16.2 to 17.1, you must include the YAML file that you created or updated during the upgrade process in the openstack overcloud node provision command. For example, use the /home/stack/tripleo-[stack]-baremetal-deploy.yaml file instead of the /home/stack/templates/overcloud-baremetal-deployed.yaml file. For more information, see Performing the overcloud adoption and preparation in Framework for upgrades (16.2 to 17.1) . Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active : Add the generated overcloud-baremetal-deployed.yaml file to the stack with your other environment files and deploy the overcloud: 10.3. Scaling down bare-metal nodes To scale down the number of bare-metal nodes in your overcloud, tag the nodes that you want to delete from the stack in the node definition file, redeploy the overcloud, and then delete the bare-metal node from the overcloud. Prerequisites A successful undercloud installation. For more information, see Installing director on the undercloud . A successful overcloud deployment. For more information, see Configuring a basic overcloud with pre-provisioned nodes . If you are replacing an Object Storage node, replicate data from the node you are removing to the new replacement node. Wait for a replication pass to finish on the new node. Check the replication pass progress in the /var/log/swift/swift.log file. When the pass finishes, the Object Storage service (swift) adds entries to the log similar to the following example: Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Decrement the count parameter in the overcloud-baremetal-deploy.yaml file, for the roles that you want to scale down. Define the hostname and name of each node that you want to remove from the stack, if they are not already defined in the instances attribute for the role. Add the attribute provisioned: false to the node that you want to remove. For example, to remove the node overcloud-objectstorage-1 from the stack, include the following snippet in your overcloud-baremetal-deploy.yaml file: After you redeploy the overcloud, the nodes that you define with the provisioned: false attribute are no longer present in the stack. However, these nodes are still running in a provisioned state. Note If you want to remove a node from the stack temporarily, after you deploy the overcloud with the attribute provisioned: false , you can then redeploy the overcloud with the attribute provisioned: true to return the node to the stack. Delete the node from the overcloud: Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud . Note Do not include the nodes that you want to remove from the stack as command arguments in the openstack overcloud node delete command. Delete the ironic node: Replace <ironic_node_uuid> with the UUID of the node. Provision the overcloud nodes to generate an updated heat environment file for inclusion in the deployment command: Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml . Add the overcloud-baremetal-deployed.yaml file generated by the provisioning command to the stack with your other environment files, and deploy the overcloud: 10.4. Scaling up pre-provisioned nodes When scaling up the overcloud with pre-provisioned nodes, you must configure the orchestration agent on each node to correspond to the director node count. Procedure Prepare the new pre-provisioned nodes. For more information, see Pre-provisioned node requirements . Scale up the nodes. For more information, see Scaling overcloud nodes . 10.5. Scaling down pre-provisioned nodes When scaling down an overcloud that has pre-provisioned nodes, follow the scale down instructions in Scaling overcloud nodes . In scale-down operations, you can use host names for both Red Hat OpenStack Platform (RHOSP) provisioned or pre-provisioned nodes. You can also use the UUID for RHOSP provisioned nodes. However, there is no UUID for pre-provisoned nodes, so you always use the host name. Procedure Retrieve the names of the nodes that you want to remove: Delete the nodes: Replace <overcloud> with the name or UUID of the overcloud stack. Replace <node> with the host names of the nodes that you want to remove, retrieved from the stack_name column returned in step 1. Ensure that the node is deleted: The status of the overcloud stack shows UPDATE_COMPLETE when the delete operation is complete. Power off the removed nodes. In a standard deployment, the bare-metal services on director power off the removed nodes. With pre-provisioned nodes, you must either manually shut down the removed nodes or use the power management control for each physical system. If you do not power off the nodes after removing them from the stack, they might remain operational and reconnect as part of the overcloud environment. Re-provision the removed nodes to a base operating system configuration so that they do not unintentionally join the overcloud in the future. Note Do not attempt to reuse nodes previously removed from the overcloud without first reprovisioning them with a fresh base operating system. The scale down process only removes the node from the overcloud stack and does not uninstall any packages. 10.6. Replacing Red Hat Ceph Storage nodes You can use director to replace Red Hat Ceph Storage nodes in a director-created cluster. For more information, see the Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director guide. 10.7. Using skip deploy identifier During a stack update operation puppet, by default, reapplies all manifests. This can result in a time consuming operation, which may not be required. To override the default operation, use the skip-deploy-identifier option. Use this option if you do not want the deployment command to generate a unique identifier for the DeployIdentifier parameter. The software configuration deployment steps only trigger if there is an actual change to the configuration. Use this option with caution and only if you are confident that you do not need to run the software configuration, such as scaling out certain roles. Note If there is a change to the puppet manifest or hierdata, puppet will reapply all manifests even when --skip-deploy-identifier is specified. 10.8. Blacklisting nodes You can exclude overcloud nodes from receiving an updated deployment. This is useful in scenarios where you want to scale new nodes and exclude existing nodes from receiving an updated set of parameters and resources from the core heat template collection. This means that the blacklisted nodes are isolated from the effects of the stack operation. Use the DeploymentServerBlacklist parameter in an environment file to create a blacklist. Setting the blacklist The DeploymentServerBlacklist parameter is a list of server names. Write a new environment file, or add the parameter value to an existing custom environment file and pass the file to the deployment command: Note The server names in the parameter value are the names according to OpenStack Orchestration (heat), not the actual server hostnames. Include this environment file with your openstack overcloud deploy command: Heat blacklists any servers in the list from receiving updated heat deployments. After the stack operation completes, any blacklisted servers remain unchanged. You can also power off or stop the os-collect-config agents during the operation. Warning Exercise caution when you blacklist nodes. Only use a blacklist if you fully understand how to apply the requested change with a blacklist in effect. It is possible to create a hung stack or configure the overcloud incorrectly when you use the blacklist feature. For example, if cluster configuration changes apply to all members of a Pacemaker cluster, blacklisting a Pacemaker cluster member during this change can cause the cluster to fail. Do not use the blacklist during update or upgrade procedures. Those procedures have their own methods for isolating changes to particular servers. When you add servers to the blacklist, further changes to those nodes are not supported until you remove the server from the blacklist. This includes updates, upgrades, scale up, scale down, and node replacement. For example, when you blacklist existing Compute nodes while scaling out the overcloud with new Compute nodes, the blacklisted nodes miss the information added to /etc/hosts and /etc/ssh/ssh_known_hosts . This can cause live migration to fail, depending on the destination host. The Compute nodes are updated with the information added to /etc/hosts and /etc/ssh/ssh_known_hosts during the overcloud deployment where they are no longer blacklisted. Do not modify the /etc/hosts and /etc/ssh/ssh_known_hosts files manually. To modify the /etc/hosts and /etc/ssh/ssh_known_hosts files, run the overcloud deploy command as described in the Clearing the Blacklist section. Clearing the blacklist To clear the blacklist for subsequent stack operations, edit the DeploymentServerBlacklist to use an empty array: Warning Do not omit the DeploymentServerBlacklist parameter. If you omit the parameter, the overcloud deployment uses the previously saved value.
{ "nodes":[ { "mac":[ "dd:dd:dd:dd:dd:dd" ], "cpu":"4", "memory":"6144", "disk":"40", "arch":"x86_64", "pm_type":"ipmi", "pm_user":"admin", "pm_password":"p@55w0rd!", "pm_addr":"192.02.24.207" }, { "mac":[ "ee:ee:ee:ee:ee:ee" ], "cpu":"4", "memory":"6144", "disk":"40", "arch":"x86_64", "pm_type":"ipmi", "pm_user":"admin", "pm_password":"p@55w0rd!", "pm_addr":"192.02.24.208" } ] } source ~/stackrc openstack overcloud node import newnodes.json openstack overcloud node introspect --provide <node_1> [<node_2>] [<node_n>] openstack overcloud node configure <node> source ~/stackrc - name: Controller count: 3 - name: Compute count: 10 - name: ObjectStorage count: 4 - name: ObjectStorage count: 4 instances: - hostname: overcloud-objectstorage-0 name: node00 - hostname: overcloud-objectstorage-1 name: node01 - hostname: overcloud-objectstorage-2 name: node02 - hostname: overcloud-objectstorage-3 name: node03 openstack overcloud node provision --stack <stack> --network-config --output <deployment_file> /home/stack/templates/overcloud-baremetal-deploy.yaml watch openstack baremetal node list openstack overcloud deploy --templates -e [your environment files] -e /home/stack/templates/overcloud-baremetal-deployed.yaml --disable-validations Mar 29 08:49:05 localhost object-server: Object replication complete. Mar 29 08:49:11 localhost container-server: Replication run OVER Mar 29 08:49:13 localhost account-server: Replication run OVER source ~/stackrc - name: ObjectStorage count: 3 instances: - hostname: overcloud-objectstorage-0 name: node00 - hostname: overcloud-objectstorage-1 name: node01 # Removed from cluster due to disk failure provisioned: false - hostname: overcloud-objectstorage-2 name: node02 - hostname: overcloud-objectstorage-3 name: node03 openstack overcloud node delete --stack <stack> --baremetal-deployment /home/stack/templates/overcloud-baremetal-deploy.yaml openstack baremetal node delete <ironic_node_uuid> openstack overcloud node provision --stack <stack> --output <deployment_file> /home/stack/templates/overcloud-baremetal-deploy.yaml openstack overcloud deploy -e /usr/share/openstack-tripleo-heat-templates/environments -e /home/stack/templates/overcloud-baremetal-deployed.yaml --disable-validations openstack stack resource list overcloud -n5 --filter type=OS::TripleO::ComputeDeployedServerServer openstack overcloud node delete --stack <overcloud> <node> [... <node>] openstack stack list openstack overcloud deploy --skip-deploy-identifier parameter_defaults: DeploymentServerBlacklist: - overcloud-compute-0 - overcloud-compute-1 - overcloud-compute-2 source ~/stackrc (undercloud) USD openstack overcloud deploy --templates -e server-blacklist.yaml [OTHER OPTIONS] parameter_defaults: DeploymentServerBlacklist: []