Chapter 2. Scaling data plane nodes
Chapter 2. Scaling data plane nodes You can scale out your data plane by adding new nodes to existing node sets and by adding new node sets. You can scale in your data plane by removing nodes from node sets and by removing node sets. 2.1. Adding nodes to a node set You can scale out your data plane by adding new nodes to the nodes section of an existing OpenStackDataPlaneNodeSet custom resource (CR). Prerequisites If you are adding unprovisioned nodes, then a BareMetalHost CR must be registered and inspected for each bare-metal data plane node. Each bare-metal node must be in the Available state after inspection. For more information about configuring bare-metal nodes, see Bare metal configuration in the RHOCP Postinstallation configuration guide. Procedure Open the OpenStackDataPlaneNodeSet CR definition file for the node set you want to update, for example, openstack_data_plane.yaml . Add the new node to the node set: Pre-Provisioned: Unprovisioned: For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties in Deploying Red Hat OpenStack Services on OpenShift . Save the OpenStackDataPlaneNodeSet CR definition file. Apply the updated OpenStackDataPlaneNodeSet CR configuration: Verify that the data plane resource has been updated by confirming that the status is SetupReady : When the status is SetupReady , the command returns a condition met message, otherwise it returns a timeout error. For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift . Create a file on your workstation to define the OpenStackDataPlaneDeployment CR: Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Tip Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set. Add the OpenStackDataPlaneNodeSet CR that you modified: Save the OpenStackDataPlaneDeployment CR deployment file. Deploy the modified OpenStackDataPlaneNodeSet CR: You can view the Ansible logs while the deployment executes: If the oc logs command returns an error similar to the following error, increase the --max-log-requests value: Verify that the modified OpenStackDataPlaneNodeSet CR is deployed: For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift . If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift . If the new nodes are Compute nodes you must bring them online: Map the Compute nodes to the Compute cell that they are connected to: If you did not create additional cells, this command maps the Compute nodes to cell1 . Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane: 2.2. Adding a new node set to the data plane You can scale out your data plane by adding a new OpenStackDataPlaneNodeSet CR to the data plane. To add the new node set to an existing data plane, you must create a new OpenStackDataPlaneDeployment CR that deploys the new OpenStackDataPlaneNodeSet CR. If you want to be able to perform move operations, such as instance migration and resize, between your new node set and other node sets on your data plane, then you must also create an additional OpenStackDataPlaneDeployment CR that runs the ssh-known-hosts service on all the node sets between which move operations must be able to be performed. Procedure Create a file on your workstation to define the new OpenStackDataPlaneNodeSet CR. Define the node set. For details about how to create a node set, see one of the following procedures: Creating an OpenStackDataPlaneNodeSet CR with pre-provisioned nodes Creating an OpenStackDataPlaneNodeSet CR with unprovisioned nodes Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to deploy the new node set: Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Tip Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the new node set. Add your new OpenStackDataPlaneNodeSet CR to the list of node sets to deploy: Save the OpenStackDataPlaneDeployment CR deployment file. Deploy the new OpenStackDataPlaneNodeSet CR: You can view the Ansible logs while the deployment executes: If the oc logs command returns an error similar to the following error, increase the --max-log-requests value: Verify that the new OpenStackDataPlaneNodeSet CR is deployed: For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift . If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift . If you want to be able to migrate workloads between your new node set and other node sets on your data plane, or perform resize operations, then you must create an additional OpenStackDataPlaneDeployment CR that runs the ssh-known-hosts service on all the node sets that the move operations are expected to work between: Create a file on your workstation to define an OpenStackDataPlaneDeployment CR that enables move operations: Add the new OpenStackDataPlaneNodeSet CR to the list of node sets and all the existing OpenStackDataPlaneNodeSet CRs that move operations must be able to be performed between: Specify that only the ssh-known-hosts service is executed on the specified node sets when deploying the node sets in the move operations OpenStackDataPlaneDeployment CR: Save the OpenStackDataPlaneDeployment CR deployment file. Deploy the ssh-known-hosts service to enable move operations between the new node set and the other specified node sets on the data plane: If the new nodes are Compute nodes you must bring them online: Map the Compute nodes to the Compute cell that they are connected to: If you did not create additional cells, this command maps the Compute nodes to cell1 . Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane: 2.3. Removing a Compute node from the data plane You can remove a Compute node from a node set on the data plane. If you remove all the nodes from a node set, then you must also remove the node set from the data plane. Prerequisites You are logged in to the RHOCP cluster as a user with cluster-admin privileges. The workloads on the Compute nodes have been migrated to other Compute nodes. Procedure Access the remote shell for the openstackclient pod: Retrieve the IP address of the Compute node that you want to remove: Retrieve a list of your Compute nodes to identify the name and UUID of the node that you want to remove: Disable the nova-compute service on the Compute node to be removed: Tip Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service. Exit the OpenStackClient pod: SSH into the Compute node to be removed and stop the ovn and nova-compute containers: Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes. Replace <node_IP_address> with the IP address for the Compute node that you retrieved in step 2. Remove the systemd unit files that manage the ovn and nova-compute containers to prevent the agents from being automatically started and registered in the database if the removed node is rebooted: Disconnect from the Compute node: Access the remote shell for openstackclient : Delete the network agents for the Compute node to be removed: Delete the nova-compute service for the Compute node to be removed: Replace <node_uuid> with the UUID of the node to be removed that you retrieved in step 3. Exit the OpenStackClient pod: Remove the node from the OpenStackDataPlaneNodeSet CR: Replace <node_set_name> with the name of the OpenStackDataPlaneNodeSet CR that the node belongs to. Replace <node_name> with the name of the node defined in the nodes section of the OpenStackDataPlaneNodeSet CR. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to update the node set with the Compute node removed: Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Tip Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set. Add the OpenStackDataPlaneNodeSet CR that you removed the node from: Specify that the OpenStackDataPlaneDeployment CR should only run the ssh-known-hosts service when deploying the listed node sets: Save the OpenStackDataPlaneDeployment CR deployment file. Deploy the ssh-known-hosts service to delete the removed nodes from the known hosts lists on the remaining nodes: You can view the Ansible logs while the deployment executes: If the oc logs command returns an error similar to the following error, increase the --max-log-requests value: Verify that the modified OpenStackDataPlaneNodeSet CR is deployed: For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift . If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift . 2.4. Removing an OpenStackDataPlaneNodeSet resource You can remove a whole node set from the data plane. To remove an OpenStackDataPlaneNodeSet resource, you must perform the following tasks: Stop the ovn and nova-compute containers running on each Compute node in the node set. Disable and delete the nova-compute service from each Compute node in the node set. Delete the network agent from each Compute node in the node set. Remove the SSH host keys of the removed nodes from the nodes in the remaining node sets. Delete the node set and remove the node set from the data plane. Prerequisites You are logged in to the RHOCP cluster as a user with cluster-admin privileges. The workloads on the node set Compute nodes have been migrated to Compute nodes on another node set. Procedure Access the remote shell for the openstackclient pod: Retrieve the IP address of each Compute node you want to remove: Retrieve a list of your Compute nodes to identify the name and UUID of each node that you want to remove: Disable the nova-compute service on each Compute node to be removed: Tip Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service. Exit the OpenStackClient pod: Perform the following operations on each Compute node to be removed: SSH into the Compute node to be removed: Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes. Replace <node_IP_address> with the IP address for the Compute node that you retrieved in step 2. Stop the ovn and nova-compute containers: Mask the systemctl services that manage the ovn and nova-compute containers to prevent the agents from being automatically started and registered in the database if the removed node is rebooted: Disconnect from the Compute node: Access the remote shell for openstackclient : Delete the network agents for each Compute node in the node set: Delete the nova-compute service for each Compute node in the node set: Replace <node_uuid> with the UUID of the node to be removed that you retrieved in step 3. Exit the openstackclient pod: Search for the string secretHashes in the output of the following command for the secrets in the node set to be deleted: The secretHashes field lists all the node set secrets in key-value pair format: <key>:<value> . The following example illustrates the secretHashes format in YAML output: Delete the node set: Replace <node_set_name> with the name of the OpenStackDataPlaneNodeSet CR to be deleted. Delete the node set secrets: Replace <secret_name> with the key of the secretHashes key-value pair you retrieved in the step, for example, cert-libvirt-default-compute-4drna21w-0 . Note You can ensure that secrets created by cert-manager get removed automatically by setting the --enable-certificate-owner-ref flag for the cert-manager Operator for Red Hat OpenShift. For more information, see Deleting a TLS secret automatically upon Certificate removal . If the node set you removed listed the global ssh-known-hosts service, then you must add the ssh-known-hosts service to one of the remaining OpenStackDataPlaneNodeSet CRs listed in the OpenStackDataPlaneDeployment CR. Open the definition file for one of the remaining OpenStackDataPlaneNodeSet CRs from your workstation and add the ssh-known-hosts service to the services field in the order that it should be executed relative to the other services: Note When adding the ssh-known-hosts service to the services list in a node set definition, you must include all the required services, including the default services. If you include only the ssh-known-hosts service in the services list, then that is the only service that is deployed. Save the updated OpenStackDataPlaneNodeSet CR definition file. Apply the updated OpenStackDataPlaneNodeSet CR configuration: Create a file on your workstation to define the OpenStackDataPlaneDeployment CR that removes the node set from the data plane: Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character. Tip Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set. Add the remaining OpenStackDataPlaneNodeSet CRs in the data plane to the list of node sets to deploy: Specify that the OpenStackDataPlaneDeployment CR should only run the ssh-known-hosts service when deploying the listed node sets: Save the OpenStackDataPlaneDeployment CR deployment file. Deploy the ssh-known-hosts service to delete the removed nodes from the known hosts lists on the remaining nodes: You can view the Ansible logs while the deployment executes: If the oc logs command returns an error similar to the following error, increase the --max-log-requests value: Verify that the modified OpenStackDataPlaneNodeSet CR is deployed: For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift . If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift .
apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack-node-set spec: preProvisioned: True nodes: edpm-compute-2: hostName: edpm-compute-2 ansible: ansibleHost: 192.168.122.102 networks: - name: CtlPlane subnetName: subnet1 defaultRoute: true fixedIP: 192.168.122.102 - name: InternalApi subnetName: subnet1 - name: Storage subnetName: subnet1 - name: Tenant subnetName: subnet1 apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: openstack-node-set spec: nodes: edpm-compute-2: hostName: edpm-compute-2 oc apply -f openstack_data_plane.yaml oc wait openstackdataplanenodeset openstack-node-set --for condition=SetupReady --timeout=10m apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: <node_set_deployment_name> spec: nodeSets: - <nodeSet_name> oc create -f openstack_data_plane_deploy.yaml -n openstack oc get pod -l app=openstackansibleee -w oc logs -l app=openstackansibleee -f --max-log-requests 10 error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit oc get openstackdataplanedeployment -n openstack NAME NODESETS STATUS MESSAGE openstack-data-plane ["openstack-data-plane"] True Setup Complete oc get openstackdataplanenodeset -n openstack NAME STATUS MESSAGE openstack-data-plane True NodeSet Ready oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose oc rsh -n openstack openstackclient openstack hypervisor list apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: <node_set_deployment_name> spec: nodeSets: - <nodeSet_name> oc create -f openstack_data_plane_deploy.yaml -n openstack oc get pod -l app=openstackansibleee -w oc logs -l app=openstackansibleee -f --max-log-requests 10 error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit oc get openstackdataplanedeployment -n openstack NAME NODESETS STATUS MESSAGE openstack-data-plane ["openstack-data-plane"] True Setup Complete oc get openstackdataplanenodeset -n openstack NAME STATUS MESSAGE openstack-data-plane True NodeSet Ready apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: enable-move-operations spec: nodeSets: - enable-move-operations - - <nodeSet_name> spec: servicesOverride: - ssh-known-hosts oc create -f enable_move_operations.yaml -n openstack oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose oc rsh -n openstack openstackclient openstack hypervisor list oc rsh -n openstack openstackclient openstack hypervisor list openstack compute service list openstack compute service set <hostname> nova-compute --disable exit ssh -i <key_file_name> cloud-admin@<node_IP_address> [cloud-admin@<hostname> ~]USD sudo systemctl stop edpm_ovn_controller [cloud-admin@<hostname> ~]USD sudo systemctl stop edpm_ovn_metadata_agent [cloud-admin@<hostname> ~]USD sudo systemctl stop edpm_nova_compute [cloud-admin@<hostname> ~]USD sudo rm -f /etc/systemd/system/edpm_ovn_controller [cloud-admin@<hostname> ~]USD sudo rm -f /etc/systemd/system/edpm_ovn_metadata_agent [cloud-admin@<hostname> ~]USD sudo rm -f /etc/systemd/system/edpm_nova_compute exit oc rsh -n openstack openstackclient openstack network agent list [--host <hostname>] openstack network agent delete <agent_id> openstack compute service delete <node_uuid> exit oc patch openstackdataplanenodeset/<node_set_name> --type json --patch '[{ "op": "remove", "path": "/spec/nodes/<node_name>" }]' apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: <node_set_deployment_name> spec: nodeSets: - <nodeSet_name> spec: servicesOverride: - ssh-known-hosts oc create -f openstack_data_plane_deploy.yaml -n openstack oc get pod -l app=openstackansibleee -w oc logs -l app=openstackansibleee -f --max-log-requests 10 error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit oc get openstackdataplanedeployment -n openstack NAME NODESETS STATUS MESSAGE openstack-data-plane ["openstack-data-plane"] True Setup Complete oc get openstackdataplanenodeset -n openstack NAME STATUS MESSAGE openstack-data-plane True NodeSet Ready oc rsh -n openstack openstackclient openstack hypervisor list openstack compute service list openstack compute service set <hostname> nova-compute --disable exit ssh -i <key_file_name> cloud-admin@<node_IP_address> [cloud-admin@<hostname> ~]USD sudo systemctl stop edpm_ovn_controller [cloud-admin@<hostname> ~]USD sudo systemctl stop edpm_ovn_metadata_agent [cloud-admin@<hostname> ~]USD sudo systemctl stop edpm_nova_compute [cloud-admin@<hostname> ~]USD sudo systemctl mask edpm_ovn_controller [cloud-admin@<hostname> ~]USD sudo systemctl mask edpm_ovn_metadata_agent [cloud-admin@<hostname> ~]USD sudo systemctl mask edpm_nova_compute exit oc rsh -n openstack openstackclient openstack network agent list [--host <hostname>] openstack network agent delete <agent_id> openstack compute service delete <node_uuid> exit oc get openstackdataplanenodeset <node_set_name> -n openstack -o yaml secretHashes: cert-libvirt-default-compute-4drna21w-0: n68chbfh678h5dfhcfh576h546h566h5c4h5cdh679hffh67h79h98h56fhc4h588h58fhb4h548h59fh554h54fh5cdh646h577hffhbdh569h5f9h68bq cert-libvirt-default-compute-4drna21w-1: n68dhdbh65chdbh5f7h695h7h54chcdh654h59ch564h5d6hdch66h54bh66ch556h649h666h76h55hc7h564h65dh5fch5c7h5fbh8bh55hcbh5b5q oc delete openstackdataplanenodeset/<node_set_name> -n openstack oc delete secret <secret_name> apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneNodeSet metadata: name: <node_set_name> spec: services: - download-cache - bootstrap - configure-network - validate-network - install-os - configure-os - ssh-known-hosts - run-os - libvirt - nova - ovn - neutron-metadata - telemetry oc apply -f <node_set_name>.yaml apiVersion: dataplane.openstack.org/v1beta1 kind: OpenStackDataPlaneDeployment metadata: name: <node_set_deployment_name> spec: nodeSets: - <nodeSet_name> spec: servicesOverride: - ssh-known-hosts oc create -f openstack_data_plane_deploy.yaml -n openstack oc get pod -l app=openstackansibleee -w oc logs -l app=openstackansibleee -f --max-log-requests 10 error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit oc get openstackdataplanedeployment -n openstack NAME NODESETS STATUS MESSAGE openstack-data-plane ["openstack-data-plane"] True Setup Complete oc get openstackdataplanenodeset -n openstack NAME STATUS MESSAGE openstack-data-plane True NodeSet Ready