Chapter 11. Troubleshooting and maintaining the Load-balancing service
Chapter 11. Troubleshooting and maintaining the Load-balancing service Basic troubleshooting and maintenance for the Red Hat OpenStack Services on OpenShift (RHOSO) Load-balancing service (octavia) starts with being familiar with the OpenStack client commands for showing status and migrating instances, and knowing how to access logs. If you need to troubleshoot more in depth, you can SSH into one or more Load-balancing service instances (amphorae). Section 11.1, "Verifying the load balancer" Section 11.3, "Showing listener statistics" Section 11.4, "Interpreting listener request errors" Section 11.5, "Viewing the load-balancing management network" 11.1. Verifying the load balancer You can troubleshoot the Red Hat OpenStack Services on OpenShift (RHOSO) Load-balancing service (octavia) and its various components by viewing the output of the load balancer show and list commands. Prerequisites The administrator has created a project for you and has provided you with a clouds.yaml file for you to access the cloud. The python-openstackclient package resides on your workstation. Procedure Confirm that the system OS_CLOUD variable is set for your cloud: USD echo USDOS_CLOUD my_cloud Reset the variable if necessary: USD export OS_CLOUD=my_other_cloud As an alternative, you can specify the cloud name by adding the --os-cloud <cloud_name> option each time you run an openstack command. Verify the load balancer ( lb1 ) settings. Note Values inside parentheses are sample values that are used in the example commands in this procedure. Substitute these sample values with values that are appropriate for your site. Example Sample output Using the loadbalancer ID ( 265d0b71-c073-40f4-9718-8a182c6d53ca ) from the step, obtain the ID of the amphora associated with the load balancer ( lb1 ). Example Sample output Using the amphora ID ( 1afabefd-ba09-49e1-8c39-41770aa25070 ) from the step, view amphora information. Example Sample output View the listener ( listener1 ) details. Example Sample output View the pool ( pool1 ) and load-balancer members. Example Sample output Verify HTTPS traffic flows across a load balancer whose listener is configured for HTTPS or TERMINATED_HTTPS protocols by connecting to the VIP address ( 192.0.2.177 ) of the load balancer. Tip Obtain the load-balancer VIP address by using the command, openstack loadbalancer show <load_balancer_name> . Note Security groups implemented for the load balancer VIP only allow data traffic for the required protocols and ports. For this reason you cannot ping load balancer VIPs, because ICMP traffic is blocked. Example Sample output 11.2. Migrating a specific Load-balancing service instance In some cases you must migrate a Load-balancing service instance (amphora). For example, if the host is being shut down for maintenance. Prerequisites You have the oc command line tool installed on your workstation. You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Locate the ID of the amphora that you want to migrate. You need to provide the ID in a later step. To prevent the Compute scheduler service from scheduling any new amphorae to the Compute node being evacuated, disable the Compute node ( compute-host-1 ). Note Values inside parentheses are sample values that are used in the example commands in this procedure. Substitute these sample values with values that are appropriate for your site. Example Fail over the amphora by using the amphora ID ( ea17210a-1076-48ff-8a1f-ced49ccb5e53 ) that you obtained. Example Exit the openstackclient pod: USD exit 11.3. Showing listener statistics Using the OpenStack Client, you can obtain statistics about the listener for a particular Red Hat OpenStack Services on OpenShift (RHOSO) loadbalancer: current active connections ( active_connections ). total bytes received ( bytes_in ). total bytes sent ( bytes_out ). total requests that were unable to be fulfilled ( request_errors ). total connections handled ( total_connections ). Prerequisites The administrator has created a project for you and has provided you with a clouds.yaml file for you to access the cloud. The python-openstackclient package resides on your workstation. Procedure Confirm that the system OS_CLOUD variable is set for your cloud: USD echo USDOS_CLOUD my_cloud Reset the variable if necessary: USD export OS_CLOUD=my_other_cloud As an alternative, you can specify the cloud name by adding the --os-cloud <cloud_name> option each time you run an openstack command. View the stats for the listener ( listener1 ). Note Values inside parentheses are sample values that are used in the example commands in this procedure. Substitute these sample values with values that are appropriate for your site. Example Tip If you do not know the name of the listener, enter the command loadbalancer listener list . Sample output Additional resources Section 11.4, "Interpreting listener request errors" 11.4. Interpreting listener request errors You can obtain statistics about the listener for a particular Red Hat OpenStack Services on OpenShift (RHOSO) loadbalancer. For more information, see Section 11.3, "Showing listener statistics" . One of the statistics tracked by the RHOSO loadbalancer, request_errors , is only counting errors that occurred in the request from the end user connecting to the load balancer. The request_errors variable is not measuring errors reported by the member server. For example, if a tenant connects through the RHOSO Load-balancing service (octavia) to a web server that returns an HTTP status code of 400 (Bad Request) , this error is not collected by the Load-balancing service. Loadbalancers do not inspect the content of data traffic. In this example, the loadbalancer interprets this flow as successful because it transported information between the user and the web server correctly. The following conditions can cause the request_errors variable to increment: early termination from the client, before the request has been sent. read error from the client. client timeout. client closed the connection. various bad requests from the client. Additional resources Section 11.3, "Showing listener statistics" 11.5. Viewing the load-balancing management network After the octavia operator has finished deploying octavia, you can view details about the management network. Procedure Run the following OpenStack client command using the oc rsh command: Sample output - ID: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b Name: octavia-provider-net Subnets: - eea45073-6e56-47fd-9153-12f7f49bc115 - ID: 77881d3f-04b0-46cb-931f-d54003cce9f0 Name: lb-mgmt-net Subnets: - e4ab96af-8077-4971-baa4-e0d40a16f55a The octavia-provider-net is the external provider network and uses the octavia network attachment interface as the physical network. Linked to the octavia network attachment. This network is limited to the OpenShift control plane. lb-mgmt-net is a self-serve tenant network that the connects the Octavia amphora instances. The amphora controllers do not have direct access to the lb-mgmt-net network. It is accessed through the octavia network attachment and a router that the octavia-operator manages. The subnets can be viewed by running oc rsh openstackclient subnet list -f yaml : - ID: e4ab96af-8077-4971-baa4-e0d40a16f55a Name: lb-mgmt-subnet Network: 77881d3f-04b0-46cb-931f-d54003cce9f0 Subnet: 172.24.0.0/16 - ID: eea45073-6e56-47fd-9153-12f7f49bc115 Name: octavia-provider-subnet Network: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b Subnet: 172.23.0.0/24 The subnet CIDR for octavia-provider-subnet is taken from the octavia network attachment and the Subnet CIDR of lb-mgmt-subnet is taken from the dst field of the octavia network attachment routes. The octavia-link-router handles the routing between the octavia-provider-net and lb-mgmt-net networks. To view the routers run oc rsh openstackclient openstack router list -f yaml : - ID: 371d800c-c803-4210-836b-eb468654462a Name: octavia-link-router Project: dc65b54e9cba475ba0adba7f898060f2 State: true Status: ACTIVE The details of the octavia-link-router reveal how it is configured to treat the networks. These can be retrieved by running oc rsh openstackclient openstack router show -f yaml octavia-link-router : admin_state_up: true availability_zone_hints: [] availability_zones: [] created_at: '2024-06-11T17:20:57Z' description: '' enable_ndp_proxy: null external_gateway_info: enable_snat: false external_fixed_ips: - ip_address: 172.23.0.150 subnet_id: eea45073-6e56-47fd-9153-12f7f49bc115 network_id: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b flavor_id: null id: 371d800c-c803-4210-836b-eb468654462a interfaces_info: - ip_address: 172.24.1.89 port_id: 1a44e94d-f44a-4752-81db-bc5402857a08 subnet_id: e4ab96af-8077-4971-baa4-e0d40a16f55a name: octavia-link-router project_id: dc65b54e9cba475ba0adba7f898060f2 revision_number: 4 routes: [] status: ACTIVE tags: [] tenant_id: dc65b54e9cba475ba0adba7f898060f2 updated_at: '2024-06-11T17:21:01Z' The external_gateway_info of the router will correspond to the gw field of the routes provided in the network attachment. Also notice that source network address translation is disabled. This is important as the amphora controllers communicate with the amphora using the addresses on the lb-mgmt-net that OpenStack allocates, not a floating IP. The routes of the network attachment direct traffic from the amphora controllers to the router and the host routes on the lb-mgmt-net subnet establish the reverse route. This host route will use the ip_address of the port in interfaces_info as the next_hop and the Subnet of the octavia-provider-subnet as the Destination to be routed to. To view the host routes for the lb-mgmt-subnet , run oc rsh openstackclient openstack subnet show lb-mgmt-subnet -c host_routes -f yaml host_routes: - destination: 172.23.0.0/24 nexthop: 172.24.1.89 The port used to connect lb-mgmt-subnet to the router is named lb-mgmt-router-port and the details can be viewed by running oc rsh openstackclient openstack port show lb-mgmt-router-port -f yaml . Note that the port_id in the router's interface_info can be used instead of the port name. admin_state_up: true allowed_address_pairs: [] binding_host_id: '' binding_profile: {} binding_vif_details: {} binding_vif_type: unbound binding_vnic_type: normal created_at: '2024-06-11T17:20:41Z' data_plane_status: null description: '' device_id: 371d800c-c803-4210-836b-eb468654462a device_owner: network:router_interface device_profile: null dns_assignment: - fqdn: host-172-24-1-89.openstackgate.local. hostname: host-172-24-1-89 ip_address: 172.24.1.89 dns_domain: '' dns_name: '' extra_dhcp_opts: [] fixed_ips: - ip_address: 172.24.1.89 subnet_id: e4ab96af-8077-4971-baa4-e0d40a16f55a id: 1a44e94d-f44a-4752-81db-bc5402857a08 ip_allocation: immediate mac_address: fa:16:3e:ba:be:ee name: lb-mgmt-router-port network_id: 77881d3f-04b0-46cb-931f-d54003cce9f0 numa_affinity_policy: null port_security_enabled: true project_id: dc65b54e9cba475ba0adba7f898060f2 propagate_uplink_status: null qos_network_policy_id: null qos_policy_id: null resource_request: null revision_number: 3 security_group_ids: - 055686ce-fb2d-409b-ab74-85df9ab3a9e0 - 5c41444b-0863-4609-9335-d5a66bdbcad8 status: ACTIVE tags: [] trunk_details: null updated_at: '2024-06-11T17:21:03Z' The fixed_ips , device_id and device_owner are all of interest: fixed_ips will match the IP for the interfaces_info of the octavia-link-router device_id will match the ID for the octavia-link-router device_owner indicates that OpenStack is using the port as a router interface
dnf list installed python-openstackclient echo USDOS_CLOUD my_cloud export OS_CLOUD=my_other_cloud openstack loadbalancer show lb1 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2024-02-17T15:59:18 | | description | | | flavor_id | None | | id | 265d0b71-c073-40f4-9718-8a182c6d53ca | | listeners | 5aaa67da-350d-4125-9022-238e0f7b7f6f | | name | lb1 | | operating_status | ONLINE | | pools | 48f6664c-b192-4763-846a-da568354da4a | | project_id | 52376c9c5c2e434283266ae7cacd3a9c | | provider | amphora | | provisioning_status | ACTIVE | | updated_at | 2024-02-17T16:01:21 | | vip_address | 192.0.2.177 | | vip_network_id | afeaf55e-7128-4dff-80e2-98f8d1f2f44c | | vip_port_id | 94a12275-1505-4cdc-80c9-4432767a980f | | vip_qos_policy_id | None | | vip_subnet_id | 06ffa90e-2b86-4fe3-9731-c7839b0be6de | +---------------------+--------------------------------------+ openstack loadbalancer amphora list | grep 265d0b71-c073-40f4-9718-8a182c6d53ca | 1afabefd-ba09-49e1-8c39-41770aa25070 | 265d0b71-c073-40f4-9718-8a182c6d53ca | ALLOCATED | STANDALONE | 198.51.100.7 | 192.0.2.177 | openstack loadbalancer amphora show 1afabefd-ba09-49e1-8c39-41770aa25070 +-----------------+--------------------------------------+ | Field | Value | +-----------------+--------------------------------------+ | id | 1afabefd-ba09-49e1-8c39-41770aa25070 | | loadbalancer_id | 265d0b71-c073-40f4-9718-8a182c6d53ca | | compute_id | ba9fc1c4-8aee-47ad-b47f-98f12ea7b200 | | lb_network_ip | 198.51.100.7 | | vrrp_ip | 192.0.2.36 | | ha_ip | 192.0.2.177 | | vrrp_port_id | 07dcd894-487a-48dc-b0ec-7324fe5d2082 | | ha_port_id | 94a12275-1505-4cdc-80c9-4432767a980f | | cert_expiration | 2026-03-19T15:59:23 | | cert_busy | False | | role | STANDALONE | | status | ALLOCATED | | vrrp_interface | None | | vrrp_id | 1 | | vrrp_priority | None | | cached_zone | nova | | created_at | 2024-02-17T15:59:22 | | updated_at | 2024-02-17T16:00:50 | | image_id | 53001253-5005-4891-bb61-8784ae85e962 | | compute_flavor | 65 | +-----------------+--------------------------------------+ openstack loadbalancer listener show listener1 +-----------------------------+--------------------------------------+ | Field | Value | +-----------------------------+--------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2024-02-17T16:00:59 | | default_pool_id | 48f6664c-b192-4763-846a-da568354da4a | | default_tls_container_ref | None | | description | | | id | 5aaa67da-350d-4125-9022-238e0f7b7f6f | | insert_headers | None | | l7policies | | | loadbalancers | 265d0b71-c073-40f4-9718-8a182c6d53ca | | name | listener1 | | operating_status | ONLINE | | project_id | 52376c9c5c2e434283266ae7cacd3a9c | | protocol | HTTP | | protocol_port | 80 | | provisioning_status | ACTIVE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2024-02-17T16:01:21 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+--------------------------------------+ openstack loadbalancer pool show pool1 +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2024-02-17T16:01:08 | | description | | | healthmonitor_id | 4b24180f-74c7-47d2-b0a2-4783ada9a4f0 | | id | 48f6664c-b192-4763-846a-da568354da4a | | lb_algorithm | ROUND_ROBIN | | listeners | 5aaa67da-350d-4125-9022-238e0f7b7f6f | | loadbalancers | 265d0b71-c073-40f4-9718-8a182c6d53ca | | members | b92694bd-3407-461a-92f2-90fb2c4aedd1 | | | 4ccdd1cf-736d-4b31-b67c-81d5f49e528d | | name | pool1 | | operating_status | ONLINE | | project_id | 52376c9c5c2e434283266ae7cacd3a9c | | protocol | HTTP | | provisioning_status | ACTIVE | | session_persistence | None | | updated_at | 2024-02-17T16:05:21 | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | +----------------------+--------------------------------------+ curl -v https://192.0.2.177 --insecure * About to connect() to 192.0.2.177 port 443 (#0) * Trying 192.0.2.177 * Connected to 192.0.2.177 (192.0.2.177) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * skipping SSL peer certificate verification * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 * Server certificate: * subject: CN=www.example.com,O=Dis,L=Springfield,ST=Denial,C=US * start date: Jan 15 09:21:45 2024 GMT * expire date: Jan 15 09:21:45 2027 GMT * common name: www.example.com * issuer: CN=www.example.com,O=Dis,L=Springfield,ST=Denial,C=US > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 192.0.2.177 > Accept: */* > < HTTP/1.1 200 OK < Content-Length: 30 < * Connection #0 to host 192.0.2.177 left intact oc rsh -n openstack openstackclient openstack loadbalancer amphora list openstack compute service set compute-host-1 nova-compute --disable openstack loadbalancer amphora failover ea17210a-1076-48ff-8a1f-ced49ccb5e53 exit dnf list installed python-openstackclient echo USDOS_CLOUD my_cloud export OS_CLOUD=my_other_cloud openstack loadbalancer listener stats show listener1 +--------------------+-------+ | Field | Value | +--------------------+-------+ | active_connections | 0 | | bytes_in | 0 | | bytes_out | 0 | | request_errors | 0 | | total_connections | 0 | +--------------------+-------+ oc rsh openstack client openstack network list -f yaml - ID: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b Name: octavia-provider-net Subnets: - eea45073-6e56-47fd-9153-12f7f49bc115 - ID: 77881d3f-04b0-46cb-931f-d54003cce9f0 Name: lb-mgmt-net Subnets: - e4ab96af-8077-4971-baa4-e0d40a16f55a - ID: e4ab96af-8077-4971-baa4-e0d40a16f55a Name: lb-mgmt-subnet Network: 77881d3f-04b0-46cb-931f-d54003cce9f0 Subnet: 172.24.0.0/16 - ID: eea45073-6e56-47fd-9153-12f7f49bc115 Name: octavia-provider-subnet Network: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b Subnet: 172.23.0.0/24 - ID: 371d800c-c803-4210-836b-eb468654462a Name: octavia-link-router Project: dc65b54e9cba475ba0adba7f898060f2 State: true Status: ACTIVE admin_state_up: true availability_zone_hints: [] availability_zones: [] created_at: '2024-06-11T17:20:57Z' description: '' enable_ndp_proxy: null external_gateway_info: enable_snat: false external_fixed_ips: - ip_address: 172.23.0.150 subnet_id: eea45073-6e56-47fd-9153-12f7f49bc115 network_id: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b flavor_id: null id: 371d800c-c803-4210-836b-eb468654462a interfaces_info: - ip_address: 172.24.1.89 port_id: 1a44e94d-f44a-4752-81db-bc5402857a08 subnet_id: e4ab96af-8077-4971-baa4-e0d40a16f55a name: octavia-link-router project_id: dc65b54e9cba475ba0adba7f898060f2 revision_number: 4 routes: [] status: ACTIVE tags: [] tenant_id: dc65b54e9cba475ba0adba7f898060f2 updated_at: '2024-06-11T17:21:01Z' host_routes: - destination: 172.23.0.0/24 nexthop: 172.24.1.89 admin_state_up: true allowed_address_pairs: [] binding_host_id: '' binding_profile: {} binding_vif_details: {} binding_vif_type: unbound binding_vnic_type: normal created_at: '2024-06-11T17:20:41Z' data_plane_status: null description: '' device_id: 371d800c-c803-4210-836b-eb468654462a device_owner: network:router_interface device_profile: null dns_assignment: - fqdn: host-172-24-1-89.openstackgate.local. hostname: host-172-24-1-89 ip_address: 172.24.1.89 dns_domain: '' dns_name: '' extra_dhcp_opts: [] fixed_ips: - ip_address: 172.24.1.89 subnet_id: e4ab96af-8077-4971-baa4-e0d40a16f55a id: 1a44e94d-f44a-4752-81db-bc5402857a08 ip_allocation: immediate mac_address: fa:16:3e:ba:be:ee name: lb-mgmt-router-port network_id: 77881d3f-04b0-46cb-931f-d54003cce9f0 numa_affinity_policy: null port_security_enabled: true project_id: dc65b54e9cba475ba0adba7f898060f2 propagate_uplink_status: null qos_network_policy_id: null qos_policy_id: null resource_request: null revision_number: 3 security_group_ids: - 055686ce-fb2d-409b-ab74-85df9ab3a9e0 - 5c41444b-0863-4609-9335-d5a66bdbcad8 status: ACTIVE tags: [] trunk_details: null updated_at: '2024-06-11T17:21:03Z'