Chapter 2. Logging
Chapter 2. Logging Red Hat OpenStack Platform writes informational messages to specific log files; you can use these messages for troubleshooting and monitoring system events. Note You do not need to attach the individual log files to your support cases manually. All the required information is gathered automatically by the sosreport utility, which is described in Chapter 5, Troubleshooting . 2.1. Log files for OpenStack services Each OpenStack component has a separate logging directory containing files specific to a running service. 2.1.1. Bare Metal Provisioning (ironic) Log Files Service Service Name Log Path OpenStack Ironic API openstack-ironic-api.service /var/log/containers/ironic/ironic-api.log OpenStack Ironic Conductor openstack-ironic-conductor.service /var/log/containers/ironic/ironic-conductor.log 2.1.2. Block Storage (cinder) Log Files Service Service Name Log Path Block Storage API openstack-cinder-api.service /var/log/containers/cinder-api.log Block Storage Backup openstack-cinder-backup.service /var/log/containers/cinder/backup.log Informational messages The cinder-manage command /var/log/containers/cinder/cinder-manage.log Block Storage Scheduler openstack-cinder-scheduler.service /var/log/containers/cinder/scheduler.log Block Storage Volume openstack-cinder-volume.service /var/log/containers/cinder/volume.log 2.1.3. Compute (nova) Log Files Service Service Name Log Path OpenStack Compute API service openstack-nova-api.service /var/log/containers/nova/nova-api.log OpenStack Compute certificate server openstack-nova-cert.service /var/log/containers/nova/nova-cert.log OpenStack Compute service openstack-nova-compute.service /var/log/containers/nova/nova-compute.log OpenStack Compute Conductor service openstack-nova-conductor.service /var/log/containers/nova/nova-conductor.log OpenStack Compute VNC console authentication server openstack-nova-consoleauth.service /var/log/containers/nova/nova-consoleauth.log Informational messages nova-manage command /var/log/containers/nova/nova-manage.log OpenStack Compute NoVNC Proxy service openstack-nova-novncproxy.service /var/log/containers/nova/nova-novncproxy.log OpenStack Compute Scheduler service openstack-nova-scheduler.service /var/log/containers/nova/nova-scheduler.log 2.1.4. Dashboard (horizon) Log Files Service Service Name Log Path Log of certain user interactions Dashboard interface /var/log/containers/horizon/horizon.log The Apache HTTP server uses several additional log files for the Dashboard web interface, which can be accessed using a web browser or command-line clients (keystone, nova). The following log files can be helpful in tracking the usage of the Dashboard and diagnosing faults: Purpose Log Path All processed HTTP requests /var/log/containers/httpd/horizon_access.log HTTP errors /var/log/containers/httpd/horizon_error.log Admin-role API requests /var/log/containers/httpd/keystone_wsgi_admin_access.log Admin-role API errors /var/log/containers/httpd/keystone_wsgi_admin_error.log Member-role API requests /var/log/containers/httpd/keystone_wsgi_main_access.log Member-role API errors /var/log/containers/httpd/keystone_wsgi_main_error.log Note There is also /var/log/containers/httpd/default_error.log , which stores errors reported by other web services running on the same host. 2.1.5. Database as a Service (trove) Log Files Service Service Name Log Path OpenStack Trove API Service openstack-trove-api.service /var/log/containers/trove/trove-api.log OpenStack Trove Conductor Service openstack-trove-conductor.service /var/log/containers/trove/trove-conductor.log OpenStack Trove guestagent Service openstack-trove-guestagent.service /var/log/containers/trove/logfile.txt OpenStack Trove taskmanager Service openstack-trove-taskmanager.service /var/log/containers/trove/trove-taskmanager.log 2.1.6. Identity Service (keystone) Log Files Service Service Name Log Path OpenStack Identity Service openstack-keystone.service /var/log/containers/keystone/keystone.log 2.1.7. Image Service (glance) Log Files Service Service Name Log Path OpenStack Image Service API server openstack-glance-api.service /var/log/containers/glance/api.log OpenStack Image Service Registry server openstack-glance-registry.service /var/log/containers/glance/registry.log 2.1.8. Networking (neutron) Log Files Service Service Name Log Path OpenStack Neutron DHCP Agent neutron-dhcp-agent.service /var/log/containers/neutron/dhcp-agent.log OpenStack Networking Layer 3 Agent neutron-l3-agent.service /var/log/containers/neutron/l3-agent.log Metadata agent service neutron-metadata-agent.service /var/log/containers/neutron/metadata-agent.log Metadata namespace proxy n/a /var/log/containers/neutron/neutron-ns-metadata-proxy- UUID .log Open vSwitch agent neutron-openvswitch-agent.service /var/log/containers/neutron/openvswitch-agent.log OpenStack Networking service neutron-server.service /var/log/containers/neutron/server.log 2.1.9. Object Storage (swift) Log Files OpenStack Object Storage sends logs to the system logging facility only. Note By default, all Object Storage log files to /var/log/containers/swift/swift.log, using the local0, local1, and local2 syslog facilities. The log messages of Object Storage are classified into two broad categories: those by REST API services and those by background daemons. The API service messages contain one line per API request, in a manner similar to popular HTTP servers; both the frontend (Proxy) and backend (Account, Container, Object) services post such messages. The daemon messages are less structured and typically contain human-readable information about daemons performing their periodic tasks. However, regardless of which part of Object Storage produces the message, the source identity is always at the beginning of the line. An example of a proxy message: An example of ad-hoc messages from background daemons: 2.1.10. Orchestration (heat) Log Files Service Service Name Log Path OpenStack Heat API Service openstack-heat-api.service /var/log/containers/heat/heat-api.log OpenStack Heat Engine Service openstack-heat-engine.service /var/log/containers/heat/heat-engine.log Orchestration service events n/a /var/log/containers/heat/heat-manage.log 2.1.11. Shared Filesystem Service (manila) Log Files Service Service Name Log Path OpenStack Manila API Server openstack-manila-api.service /var/log/containers/manila/api.log OpenStack Manila Scheduler openstack-manila-scheduler.service /var/log/containers/manila/scheduler.log OpenStack Manila Share Service openstack-manila-share.service /var/log/containers/manila/share.log Note Some information from the Manila Python library can also be logged in /var/log/containers/manila/manila-manage.log . 2.1.12. Telemetry (ceilometer) Log Files Service Service Name Log Path OpenStack ceilometer notification agent openstack-ceilometer-notification.service /var/log/containers/ceilometer/agent-notification.log OpenStack ceilometer alarm evaluation openstack-ceilometer-alarm-evaluator.service /var/log/containers/ceilometer/alarm-evaluator.log OpenStack ceilometer alarm notification openstack-ceilometer-alarm-notifier.service /var/log/containers/ceilometer/alarm-notifier.log OpenStack ceilometer API httpd.service /var/log/containers/ceilometer/api.log Informational messages MongoDB integration /var/log/containers/ceilometer/ceilometer-dbsync.log OpenStack ceilometer central agent openstack-ceilometer-central.service /var/log/containers/ceilometer/central.log OpenStack ceilometer collection openstack-ceilometer-collector.service /var/log/containers/ceilometer/collector.log OpenStack ceilometer compute agent openstack-ceilometer-compute.service /var/log/containers/ceilometer/compute.log 2.1.13. Log Files for Supporting Services The following services are used by the core OpenStack components and have their own log directories and files. Service Service Name Log Path Message broker (RabbitMQ) rabbitmq-server.service /var/log/rabbitmq/rabbit@ short_hostname .log /var/log/rabbitmq/rabbit@ short_hostname -sasl.log (for Simple Authentication and Security Layer related log messages) Database server (MariaDB) mariadb.service /var/log/mariadb/mariadb.log Document-oriented database (MongoDB) mongod.service /var/log/mongodb/mongodb.log Virtual network switch (Open vSwitch) openvswitch-nonetwork.service /var/log/openvswitch/ovsdb-server.log /var/log/openvswitch/ovs-vswitchd.log 2.2. Enabling centralized logging during deployment To enable centralized logging, specify the implementation of the OS::TripleO::Services::Rsyslog composable service. Add the file path of the logging environment file to the overcloud deployment command, as shown in the following example: 2.3. Configuring logging features To configure logging features, modify the RsyslogElasticsearchSetting parameter in the logging-environment-rsyslog.yaml file: Copy the tripleo-heat-templates/environments/logging-environment-rsyslog.yaml file to your home directory. Create entries in the RsyslogElasticsearchSetting parameter to suit your environment. The following snippet is an example configuration of the RsyslogElasticsearchSetting parameter: Note You can configure the RsyslogElasticsearchTls parameters to enable secure data transfer. For more information, see Section 2.3.1, "Configurable parameters" . 2.3.1. Configurable parameters The following table contains descriptions of the parameters that you can configure. You can find these parameters in the tripleo-heat-templates/deployment/logging/rsyslog-container-puppet.yaml file. Parameter Description RsyslogElasticsearchSetting Configuration for rsyslog-elasticsearch plug-in. For more information, see https://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html . RsyslogElasticsearchTlsCACert Contains content of the CA cert for the CA that issued Elasticsearch server cert. RsyslogElasticsearchTlsClientCert Contains content of the client cert for doing client cert auth against Elasticsearch. RsyslogElasticsearchTlsClientKey Contains content of the private key corresponding to the cert RsyslogElasticsearchTlsClientCert . 2.4. Overriding the default path for a log file If you modify the default containers and the modification includes the path to the service log file, you must also modify the default log file path. Every composable service has a default log file path parameter with the naming convention format: <service_name>LoggingSource. For example, the parameter for nova-compute service is NovaComputeLoggingSource . To override the default path for the nova-compute service, add the path to the NovaComputeLoggingSource parameter in your configuration file. The tag and path attributes are mandatory elements of the <service_name>LoggingSource parameter. On each service, the tag and the path are defined and the rest of the values are derived by default. The following snippet is another example of overriding the default log file path: 2.5. Configuring logging options Each component maintains distinct logging configuration in the component configuration file. For example, in Compute, these options are set in /etc/nova/nova.conf : To increase the level of informational logging, enable debugging. This option increases the amount of information captured, so consider using it temporarily or first reviewing your log rotation settings. Enable verbose logging: Change the log file path: Send your logs to a central syslog server: Note You can also configure the timestamp and format of the logs. Review the configuration file of the component for additional logging options. 2.6. Testing the connection On the client side, to verify communication between Rsyslog and Elasticsearch, complete the following step: Navigate to the Elasticsearch connection log file, /var/log/rsyslog/omelasticsearch.log in the Rsyslog container or /var/log/containers/rsyslog/omelasticsearch.log on the host. If this log file does not exist or if the log file exists but does not contain logs, there is no connection problem. If the log file is present and contains logs, Rsyslog has not connected successfully. Note To test the connection from the server side, view the Elasticsearch logs for connection issues. 2.7. Server-side logging If you have an Elasticsearch cluster running, configure the RsyslogElasticsearchSetting parameter in the logging-environment-rsyslog.yaml file to connect Rsyslog that is running on overcloud nodes. To configure the RsyslogElasticsearchSetting parameter, see https://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html . 2.8. Tracebacks When you encounter an issue and you start troubleshooting, you can use a traceback log to diagnose the issue. In log files, tracebacks usually have several lines of information, all relating to the same issue. Rsyslog provides a regular expression to define how a log record starts. Each log record usually starts with a timestamp and the first line of the traceback is the only line that contains this information. Rsyslog bundles the indented records with the first line and sends them as one log record. For that behaviour configuration option startmsg.regex in <Service>LoggingSource is used. The following regular expression is the default value for all <service>LoggingSource parameters in the director: When this default does not match log records of your added or modified LoggingSource , you must change startmsg.regex accordingly.
Apr 20 15:20:34 rhev-a24c-01 proxy-server: 127.0.0.1 127.0.0.1 20/Apr/2015/19/20/34 GET /v1/AUTH_zaitcev%3Fformat%3Djson%26marker%3Dtestcont HTTP/1.0 200 - python-swiftclient-2.1.0 AUTH_tk737d6... - 2 - txc454fa8ea4844d909820a-0055355182 - 0.0162 - - 1429557634.806570053 1429557634.822791100 Apr 27 17:08:15 rhev-a24c-02 object-auditor: Object audit (ZBF). Since Mon Apr 27 21:08:15 2015: Locally: 1 passed, 0 quarantined, 0 errors files/sec: 4.34 , bytes/sec: 0.00, Total time: 0.23, Auditing time: 0.00, Rate: 0.00 Apr 27 17:08:16 rhev-a24c-02 object-auditor: Object audit (ZBF) "forever" mode completed: 0.56s. Total quarantined: 0, Total errors: 0, Total files/sec: 14.31, Total bytes/sec: 0.00, Auditing time: 0.02, Rate: 0.04 Apr 27 17:08:16 rhev-a24c-02 account-replicator: Beginning replication run Apr 27 17:08:16 rhev-a24c-02 account-replicator: Replication run OVER Apr 27 17:08:16 rhev-a24c-02 account-replicator: Attempted to replicate 5 dbs in 0.12589 seconds (39.71876/s) Apr 27 17:08:16 rhev-a24c-02 account-replicator: Removed 0 dbs Apr 27 17:08:16 rhev-a24c-02 account-replicator: 10 successes, 0 failures openstack overcloud deploy <other arguments> -e /usr/share/openstack-tripleo-heat-templates/environments/logging-environment-rsyslog.yaml parameter_defaults: RsyslogElasticsearchSetting: server: "exampleip:9200" usehttps: "on" NovaComputeLoggingSource: tag: openstack.nova.compute file: /some/other/path/nova-compute.log ServiceLoggingSource: tag: openstack.Service file: /another/path/service/service.log startmsg.regex: "^[a-zA-Z]{3} [0-9]{2} [:0-9]{8}" debug=True verbose=True log_dir=/var/log/containers/nova use_syslog=True syslog_log_facility=LOG_USER startmsg.regex='^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}(.[0-9]+ [0-9]+)? (DEBUG|INFO|WARNING|ERROR) '