Chapter 5. Managing Load-balancing service instance logs
Chapter 5. Managing Load-balancing service instance logs You can enable tenant flow logging or suppress logging to the amphora local file system. You can also forward administrative or tenant flow logs to syslog receivers at a set of containers or to other syslog receivers at endpoints of your choosing. In addition, you can control a range of other logging features such as, setting the syslog facility value, changing the tenant flow log format, and widening the scope of administrative logging to include logs from sources like the kernel and from cron. Section 5.1, "Basics of offloading Load-balancing service instance (amphora) logs" Section 5.2, "Enabling Load-balancing service instance administrative log offloading" Section 5.3, "Enabling tenant flow log offloading for Load-balancing service instances" Section 5.4, "Disabling Load-balancing service instance tenant flow logging" Section 5.5, "Disabling Load-balancing service instance local log storage" Section 5.6, "Heat parameters for Load-balancing service instance logging" Section 5.7, "Load-balancing service instance tenant flow log format" 5.1. Basics of offloading Load-balancing service instance (amphora) logs By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. However, you can specify that amphorae offload logs to syslog receivers to aggregate both administrative and tenant traffic flow logs. Log offloading enables administrators to go to one location for logs, and retain logs when amphorae are rotated. Additional resources Section 5.6, "Heat parameters for Load-balancing service instance logging" Section 5.7, "Load-balancing service instance tenant flow log format" 5.2. Enabling Load-balancing service instance administrative log offloading By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. However, you can specify that the amphorae offload logs to syslog receivers to aggregate administrative logs. Log offloading enables administrators to go to one location for logs, and retain logs when the amphorae are rotated. Procedure Log in to the undercloud host as the stack user. Source the undercloud credentials file: Create a custom YAML environment file. Example In the YAML environment file under parameter_defaults , set OctaviaLogOffload to true . Note The amphorae offload administrative logs use the syslog facility value of local1 , by default, unless you specify another value with the OctaviaAdminLogFacility parameter. Example The amphorae forward only load balancer-related administrative logs, such as the haproxy admin logs, keepalived, and amphora agent logs. If you want to configure the amphorae to send all of the administrative logs from the amphorae, such as the kernel, system, and security logs, set OctaviaForwardAllLogs to true . Example The amphorae use a set of default containers defined by the Orchestration service (heat) that contain syslog receivers listening for log messages. If you want to use a different set of endpoints, you can specify those with the OctaviaAdminLogTargets parameter: Example By default, when you enable log offloading, tenant flow logs are also offloaded. If you want to disable tenant flow log offloading, set the OctaviaConnectionLogging to false . Example Run the deployment command and include the core heat templates, environment files, and this new custom environment file. Important The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Example Verification Unless you specified specific endpoints with the OctaviaAdminLogTargets or OctaviaTenantLogTargets , the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs ( /var/log/containers/octavia/ ). Check the appropriate location for the presence of the following log files: octavia-amphora.log -- Log file for the administrative log. (if enabled) octavia-tenant-traffic.log -- Log file for the tenant traffic flow log. Additional resources Section 5.6, "Heat parameters for Load-balancing service instance logging" Environment files in the Advanced Overcloud Customization guide Including environment files in overcloud creation in the Advanced Overcloud Customization guide 5.3. Enabling tenant flow log offloading for Load-balancing service instances By default, Load-balancing service instances (amphorae) store logs on the local machine in the systemd journal. You can specify that the amphorae offload logs to syslog receivers on endpoints that contain disk space sufficient for tenant flow logs that can grow in size depending on the number of tenant connections. Tenant flow log offloading for Load-balancing service instances is automatically enabled when administrative log offloading is enabled. The only case when administrative log offloading is on and tenant flow log offloading is off, is when the OctaviaConnectionLogging parameter is set to false . Important Tenant flow logging can produce a large number of syslog messages depending on how many connections the load balancers are receiving. Tenant flow logging produces one log entry for each connection to the load balancer. Monitor log volume and configure your syslog receivers appropriately based on the expected number of connections that your load balancers manage. Procedure Log in to the undercloud host as the stack user. Source the undercloud credentials file: Locate the environment file in which the OctaviaConnectionLogging parameter is set: If you do not find the file, create an environment file: Add the OctaviaLogOffload and OctaviaConnectionLogging parameters to the parameter_defaults section of the environment file and set the values to true : Note The amphorae use the syslog facility default value of local0 to offload tenant flow logs unless you use the OctaviaTenantLogFacility parameter to specify another value. Optional: The amphorae use a set of default containers that contain syslog receivers that listen for log messages. You can change the admin and tenant log endpoints using the parameters OctaviaAdminLogTargets and OctaviaTenantLogTargets . Run the deployment command and include the core heat templates, environment files, and the custom environment file you modified. Important The order of the environment files is important because the parameters and resources defined in subsequent environment files take precedence. Verification Unless you specified specific endpoints with the OctaviaAdminLogTargets or OctaviaTenantLogTargets , the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs ( /var/log/containers/octavia/ ). Check the appropriate location for the presence of the following log files: octavia-amphora.log -- Log file for the administrative log. octavia-tenant-traffic.log -- Log file for the tenant traffic flow log. Additional resources Section 5.6, "Heat parameters for Load-balancing service instance logging" Environment files in the Advanced Overcloud Customization guide Including environment files in overcloud creation in the Advanced Overcloud Customization guide Section 5.7, "Load-balancing service instance tenant flow log format" 5.4. Disabling Load-balancing service instance tenant flow logging Tenant flow log offloading for Load-balancing service instances (amphorae) is automatically enabled when you enable administrative log offloading. To keep administrative log offloading enabled and to disable tenant flow logging, you must set the OctaviaConnectionLogging parameter to false . When the OctaviaConnectionLogging parameter is false , the amphorae do not write tenant flow logs to the disk inside the amphorae, nor offload any logs to syslog receivers listening elsewhere. Procedure Log in to the undercloud host as the stack user. Source the undercloud credentials file: Locate the YAML custom environment file in which amphora logging is configured. Example In the custom environment file, under parameter_defaults , set OctaviaConnectionLogging to false . Example Run the deployment command and include the core heat templates, environment files, and the custom environment file in which you set OctaviaConnectionLogging to true . Important The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Example Verification Unless you specified specific endpoints with the OctaviaAdminLogTargets or OctaviaTenantLogTargets , the amphorae offload logs to the RHOSP Controller in the same location as the other RHOSP logs ( /var/log/containers/octavia/ ). Check the appropriate location for the absence of octavia-tenant-traffic.log . Additional resources Environment files in the Advanced Overcloud Customization guide Including environment files in overcloud creation in the Advanced Overcloud Customization guide 5.5. Disabling Load-balancing service instance local log storage Even when you configure Load-balancing service instances (amphorae) to offload administrative and tenant flow logs, the amphorae continue to write these logs to the disk inside the amphorae. To improve the performance of the load balancer, you can stop logging locally. Important If you disable logging locally, you also disable all log storage in the amphora, including kernel, system, and security logging. Note If you disable local log storage and the OctaviaLogOffload parameter is set to false, ensure that you set OctaviaConnectionLogging to false for improved load balancing performance. Procedure Log in to the undercloud host as the stack user. Source the undercloud credentials file: Create a custom YAML environment file. Example In the environment file under parameter_defaults , set OctaviaDisableLocalLogStorage to true . Run the deployment command and include the core heat templates, environment files, and this new custom environment file. Important The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Example Verification On the amphora instance, check the location where log files are written, and verify that no new log files are being written. Additional resources Environment files in the Advanced Overcloud Customization guide Including environment files in overcloud creation in the Advanced Overcloud Customization guide 5.6. Heat parameters for Load-balancing service instance logging When you want to configure Load-balancing service instance (amphora) logging, you set values for one or more Orchestration service (heat) parameters that control logging and run the openstack overcloud deploy command. These heat parameters for amphora logging enable you to control features such as turning on log offloading, defining custom endpoints to offload logs to, setting the syslog facility value for logs, and so on. Table 5.1. Heat parameters for all logs Parameter Default Description OctaviaLogOffload false When true , instances offload their logs. If no endpoints are specified, then, by default, the instance offloads its log to the same location as the other RHOSP logs ( /var/log/containers/octavia/ ). OctaviaDisableLocalLogStorage false When true , instances do not store logs on the instance host filesystem. This includes all kernel, system, and security logs. OctaviaForwardAllLogs false When true , instances forward all log messages to the administrative log endpoints, including non-load balancing related logs such as the cron and kernel logs. For instances to recognize OctaviaForwardAllLogs , you must also enable OctaviaLogOffload . Table 5.2. Heat parameters for admin logging Parameter Default Description OctaviaAdminLogTargets No value. A comma-delimited list of syslog endpoints (<host>:<port>) to receive administrative log messages. These endpoints can be a container, VM, or physical host that is running a process that is listening for the log messages on the specified port. When OctaviaAdminLogTargets is not present, the instance offloads its log to the same location as the other RHOSP logs ( /var/log/containers/octavia/ ) on a set of containers that is defined by RHOSP director. OctaviaAdminLogFacility 1 A number between 0 and 7 that is the syslog "LOG_LOCAL" facility to use for the administrative log messages. Table 5.3. Heat parameters for tenant flow logging Parameter Default Description OctaviaConnectionLogging true When true , tenant connection flows are logged. When OctaviaConnectionLogging is false, the amphorae stop logging tenant connections regardless of the OctaviaLogOffload setting. OctaviaConnectionLogging disables local tenant flow log storage and, if log offloading is enabled, it does not forward tenant flow logs. OctaviaTenantLogTargets No value. A comma-delimited list of syslog endpoints (<host>:<port>) to receive tenant traffic flow log messages. These endpoints can be a container, VM, or physical host that is running a process that is listening for the log messages on the specified port. When OctaviaTenantLogTargets is not present, the instance offloads its log to the same location as the other RHOSP logs ( /var/log/containers/octavia/ ) on a set of containers that is defined by RHOSP director. OctaviaTenantLogFacility 0 A number between 0 and 7 that is the syslog "LOG_LOCAL" facility to use for the tenant traffic flow log messages. OctaviaUserLogFormat "{{ '{{' }} project_id {{ '}}' }} {{ '{{' }} lb_id {{ '}}' }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc" The format for the tenant traffic flow log. The alphanumerics represent specific octavia fields, and the curly braces ({}) are substitution variables. Additional resources Including environment files in overcloud creation in the Advanced Overcloud Customization guide 5.7. Load-balancing service instance tenant flow log format The log format that the tenant flow logs for Load-balancing service instances (amphorae) follows is the HAProxy log format. The two exceptions are the project_id and lb_id variables whose values are provided by the amphora provider driver. Sample Here is a sample log entry with rsyslog as the syslog receiver: Notes A hyphen (-) indicates any field that is unknown or not applicable to the connection. The prefix in the earlier sample log entry originates from the rsyslog receiver, and is not part of the syslog message from the amphora: Default The default amphora tenant flow log format is: Refer to the table that follows for a description of the format. Table 5.4. Data variables for tenant flow logs format variable definitions. Variable Type Field name {{project_id}} UUID Project ID (substitution variable from the amphora provider driver) {{lb_id}} UUID Load balancer ID (substitution variable from the amphora provider driver) %f string frontend_name %ci IP address client_ip %cp numeric client_port %t date date_time %ST numeric status_code %B numeric bytes_read %U numeric bytes_uploaded %ssl_c_verify Boolean client_certificate_verify (0 or 1) %ssl_c_s_dn string client_certificate_distinguised_name %b string pool_id %s string member_id %Tt numeric processing_time (milliseconds) %tsc string termination_state (with cookie status) Additional resources Custom log format in HAProxy Documentation
source ~/stackrc vi /home/stack/templates/my-octavia-environment.yaml parameter_defaults: OctaviaLogOffload: true parameter_defaults: OctaviaLogOffload: true OctaviaAdminLogFacility: 2 parameter_defaults: OctaviaLogOffload: true OctaviaForwardAllLogs: true OctaviaAdminLogTargets: <ip_address>:<port>[, <ip_address>:<port>] parameter_defaults: OctaviaLogOffload: true OctaviaAdminLogTargets: 192.0.2.1:10514, 2001:db8:1::10:10514 parameter_defaults: OctaviaLogOffload: true OctaviaConnectionLogging: false openstack overcloud deploy --templates -e [your-environment-files] -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml -e /home/stack/templates/my-octavia-environment.yaml source ~/stackrc grep -rl OctaviaConnectionLogging /home/stack/templates/ vi /home/stack/templates/my-octavia-environment.yaml parameter_defaults: OctaviaLogOffload: true OctaviaConnectionLogging: true OctaviaAdminLogTargets: <ip-address>:<port>[, <ip-address>:<port>] OctaviaTenantLogTargets: <ip-address>:<port>[, <ip-address>:<port>] openstack overcloud deploy --templates -e <your_environment_files> -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml -e /home/stack/templates/my-octavia-environment.yaml source ~/stackrc grep -rl OctaviaLogOffload /home/stack/templates/ parameter_defaults: OctaviaLogOffload: true OctaviaConnectionLogging: false openstack overcloud deploy --templates -e [your-environment-files] -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml -e /home/stack/templates/my-octavia-environment.yaml source ~/stackrc vi /home/stack/templates/my-octavia-environment.yaml parameter_defaults: OctaviaDisableLocalLogStorage: true openstack overcloud deploy --templates -e <your_environment_files> -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml -e /home/stack/templates/my-octavia-environment.yaml Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]: 5408b89aa45b48c69a53dca1aaec58db fd8f23df-960b-4b12-ba62-2b1dff661ee7 261ecfc2-9e8e-4bba-9ec2-3c903459a895 172.24.4.1 41152 12/Jun/2019:00:44:13.030 "GET / HTTP/1.1" 200 76 73 - "" e37e0e04-68a3-435b-876c-cffe4f2138a4 6f2720b3-27dc-4496-9039-1aafe2fee105 4 -- Jun 12 00:44:13 amphora-3e0239c3-5496-4215-b76c-6abbe18de573 haproxy[1644]:" `"{{ '{{' }} project_id {{ '}}' }} {{ '{{' }} lb_id {{ '}}' }} %f %ci %cp %t %{+Q}r %ST %B %U %[ssl_c_verify] %{+Q}[ssl_c_s_dn] %b %s %Tt %tsc"`