Chapter 7. Logging performance data with pmlogger
Chapter 7. Logging performance data with pmlogger With the PCP tool you can log the performance metric values and replay them later. This allows you to perform a retrospective performance analysis. Using the pmlogger tool, you can: Create the archived logs of selected metrics on the system Specify which metrics are recorded on the system and how often 7.1. Modifying the pmlogger configuration file with pmlogconf When the pmlogger service is running, PCP logs a default set of metrics on the host. Use the pmlogconf utility to check the default configuration. If the pmlogger configuration file does not exist, pmlogconf creates it with a default metric values. Prerequisites PCP is installed. For more information, see Installing and enabling PCP . Procedure Create or modify the pmlogger configuration file: Follow pmlogconf prompts to enable or disable groups of related performance metrics and to control the logging interval for each enabled group. Additional resources pmlogconf(1) and pmlogger(1) man pages on your system System services and tools distributed with PCP 7.2. Editing the pmlogger configuration file manually To create a tailored logging configuration with specific metrics and given intervals, edit the pmlogger configuration file manually. The default pmlogger configuration file is /var/lib/pcp/config/pmlogger/config.default . The configuration file specifies which metrics are logged by the primary logging instance. In manual configuration, you can: Record metrics which are not listed in the automatic configuration. Choose custom logging frequencies. Add PMDA with the application metrics. Prerequisites PCP is installed. For more information, see Installing and enabling PCP . Procedure Open and edit the /var/lib/pcp/config/pmlogger/config.default file to add specific metrics: Additional resources pmlogger(1) man page on your system System services and tools distributed with PCP 7.3. Enabling the pmlogger service The pmlogger service must be started and enabled to log the metric values on the local machine. This procedure describes how to enable the pmlogger service. Prerequisites PCP is installed. For more information, see Installing and enabling PCP . Procedure Start and enable the pmlogger service: Verification Verify if the pmlogger service is enabled: Additional resources pmlogger(1) man page on your system System services and tools distributed with PCP /var/lib/pcp/config/pmlogger/config.default file 7.4. Setting up a client system for metrics collection This procedure describes how to set up a client system so that a central server can collect metrics from clients running PCP. Prerequisites PCP is installed. For more information, see Installing and enabling PCP . Procedure Install the pcp-system-tools package: Configure an IP address for pmcd : Replace 192.168.4.62 with the IP address, the client should listen on. By default, pmcd is listening on the localhost. Configure the firewall to add the public zone permanently: Set an SELinux boolean: Enable the pmcd and pmlogger services: Verification Verify if the pmcd is correctly listening on the configured IP address: Additional resources pmlogger(1) , firewall-cmd(1) , ss(8) , and setsebool(8) man pages on your system System services and tools distributed with PCP /var/lib/pcp/config/pmlogger/config.default file 7.5. Setting up a central server to collect data This procedure describes how to create a central server to collect metrics from clients running PCP. Prerequisites PCP is installed. For more information, see Installing and enabling PCP . Client is configured for metrics collection. For more information, see Setting up a client system for metrics collection . Procedure Install the pcp-system-tools package: Create the /etc/pcp/pmlogger/control.d/remote file with the following content: Replace 192.168.4.13 , 192.168.4.14 and 192.168.4.62 with the client IP addresses. Note In Red Hat Enterpirse Linux 8.0, 8.1 and 8.2 use the following format for remote hosts in the control file: PCP_LOG_DIR/pmlogger/ host_name . Enable the pmcd and pmlogger services: Verification Ensure that you can access the latest archive file from each directory: The archive files from the /var/log/pcp/pmlogger/ directory can be used for further analysis and graphing. Additional resources pmlogger(1) man page on your system System services and tools distributed with PCP /var/lib/pcp/config/pmlogger/config.default file 7.6. Systemd units and pmlogger When you deploy the pmlogger service, either as a single host monitoring itself or a pmlogger farm with a single host collecting metrics from several remote hosts, there are several associated systemd service and timer units that are automatically deployed. These services and timers provide routine checks to ensure that your pmlogger instances are running, restart any missing instances, and perform archive management such as file compression. The checking and housekeeping services typically deployed by pmlogger are: pmlogger_daily.service Runs daily, soon after midnight by default, to aggregate, compress, and rotate one or more sets of PCP archives. Also culls archives older than the limit, 2 weeks by default. Triggered by the pmlogger_daily.timer unit, which is required by the pmlogger.service unit. pmlogger_check Performs half-hourly checks that pmlogger instances are running. Restarts any missing instances and performs any required compression tasks. Triggered by the pmlogger_check.timer unit, which is required by the pmlogger.service unit. pmlogger_farm_check Checks the status of all configured pmlogger instances. Restarts any missing instances. Migrates all non-primary instances to the pmlogger_farm service. Triggered by the pmlogger_farm_check.timer , which is required by the pmlogger_farm.service unit that is itself required by the pmlogger.service unit. These services are managed through a series of positive dependencies, meaning that they are all enabled upon activating the primary pmlogger instance. Note that while pmlogger_daily.service is disabled by default, pmlogger_daily.timer being active via the dependency with pmlogger.service will trigger pmlogger_daily.service to run. pmlogger_daily is also integrated with pmlogrewrite for automatically rewriting archives before merging. This helps to ensure metadata consistency amid changing production environments and PMDAs. For example, if pmcd on one monitored host is updated during the logging interval, the semantics for some metrics on the host might be updated, thus making the new archives incompatible with the previously recorded archives from that host. For more information see the pmlogrewrite(1) man page. Managing systemd services triggered by pmlogger You can create an automated custom archive management system for data collected by your pmlogger instances. This is done using control files. These control files are: For the primary pmlogger instance: etc/pcp/pmlogger/control /etc/pcp/pmlogger/control.d/local For the remote hosts: /etc/pcp/pmlogger/control.d/remote Replace remote with your desired file name. NOTE The primary pmlogger instance must be running on the same host as the pmcd it connects to. You do not need to have a primary instance and you might not need it in your configuration if one central host is collecting data on several pmlogger instances connected to pmcd instances running on remote host The file should contain one line for each host to be logged. The default format of the primary logger instance that is automatically created looks similar to: The fields are: Host The name of the host to be logged P? Stands for "Primary?" This field indicates if the host is the primary logger instance, y , or not, n . There can only be one primary logger across all the files in your configuration and it must be running on the same host as the pmcd it connects to. S? Stands for "Socks?" This field indicates if this logger instance needs to use the SOCKS protocol to connect to pmcd through a firewall, y , or not, n . directory All archives associated with this line are created in this directory. args Arguments passed to pmlogger . The default values for the args field are: -r Report the archive sizes and growth rate. T24h10m Specifies when to end logging for each day. This is typically the time when pmlogger_daily.service runs. The default value of 24h10m indicates that logging should end 24 hours and 10 minutes after it begins, at the latest. -c config.default Specifies which configuration file to use. This essentially defines what metrics to record. -v 100Mb Specifies the size at which point one data volume is filled and another is created. After it switches to the new archive, the previously recorded one will be compressed by either pmlogger_daily or pmlogger_check . Additional resources pmlogger(1) and pmlogrewrite(1) man pages on your system pmlogger_daily(1) , pmlogger_check(1) , and pmlogger.control(5) man pages on your system 7.7. Replaying the PCP log archives with pmrep After recording the metric data, you can replay the PCP log archives. To export the logs to text files and import them into spreadsheets, use PCP utilities such as pcp2csv , pcp2xml , pmrep or pmlogsummary . Using the pmrep tool, you can: View the log files Parse the selected PCP log archive and export the values into an ASCII table Extract the entire archive log or only select metric values from the log by specifying individual metrics on the command line Prerequisites PCP is installed. For more information, see Installing and enabling PCP . The pmlogger service is enabled. For more information, see Enabling the pmlogger service . Install the pcp-gui package: Procedure Display the data on the metric: The mentioned example displays the data on the disk.dev.write metric collected in an archive at a 5 second interval in comma-separated-value format. Note Replace 20211128 in this example with a filename containing the pmlogger archive you want to display data for. Additional resources pmlogger(1) , pmrep(1) , and pmlogsummary(1) man pages on your system System services and tools distributed with PCP
pmlogconf -r /var/lib/pcp/config/pmlogger/config.default It is safe to make additions from here on # log mandatory on every 5 seconds { xfs.write xfs.write_bytes xfs.read xfs.read_bytes } log mandatory on every 10 seconds { xfs.allocs xfs.block_map xfs.transactions xfs.log } [access] disallow * : all; allow localhost : enquire; systemctl start pmlogger systemctl enable pmlogger pcp Performance Co-Pilot configuration on workstation: platform: Linux workstation 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 hardware: 12 cpus, 2 disks, 1 node, 36023MB RAM timezone: CEST-2 services: pmcd pmcd: Version 4.3.0-1, 8 agents, 1 client pmda: root pmcd proc xfs linux mmv kvm jbd2 pmlogger: primary logger: /var/log/pcp/pmlogger/workstation/20190827.15.54 yum install pcp-system-tools echo "-i 192.168.4.62 " >>/etc/pcp/pmcd/pmcd.options firewall-cmd --permanent --zone=public --add-port=44321/tcp success firewall-cmd --reload success setsebool -P pcp_bind_all_unreserved_ports on systemctl enable pmcd pmlogger systemctl restart pmcd pmlogger ss -tlp | grep 44321 LISTEN 0 5 127.0.0.1:44321 0.0.0.0:* users:(("pmcd",pid=151595,fd=6)) LISTEN 0 5 192.168.4.62:44321 0.0.0.0:* users:(("pmcd",pid=151595,fd=0)) LISTEN 0 5 [::1]:44321 [::]:* users:(("pmcd",pid=151595,fd=7)) yum install pcp-system-tools DO NOT REMOVE OR EDIT THE FOLLOWING LINE USDversion=1.1 192.168.4.13 n n PCP_ARCHIVE_DIR/rhel7u4a -r -T24h10m -c config.rhel7u4a 192.168.4.14 n n PCP_ARCHIVE_DIR/rhel6u10a -r -T24h10m -c config.rhel6u10a 192.168.4.62 n n PCP_ARCHIVE_DIR/rhel8u1a -r -T24h10m -c config.rhel8u1a systemctl enable pmcd pmlogger systemctl restart pmcd pmlogger for i in /var/log/pcp/pmlogger/rhel*/*.0; do pmdumplog -L USDi; done Log Label (Log Format Version 2) Performance metrics from host rhel6u10a.local commencing Mon Nov 25 21:55:04.851 2019 ending Mon Nov 25 22:06:04.874 2019 Archive timezone: JST-9 PID for pmlogger: 24002 Log Label (Log Format Version 2) Performance metrics from host rhel7u4a commencing Tue Nov 26 06:49:24.954 2019 ending Tue Nov 26 07:06:24.979 2019 Archive timezone: CET-1 PID for pmlogger: 10941 [..] === LOGGER CONTROL SPECIFICATIONS === # #Host P? S? directory args local primary logger LOCALHOSTNAME y n PCP_ARCHIVE_DIR/LOCALHOSTNAME -r -T24h10m -c config.default -v 100Mb yum install pcp-gui pmrep --start @3:00am --archive 20211128 --interval 5seconds --samples 10 --output csv disk.dev.write Time,"disk.dev.write-sda","disk.dev.write-sdb" 2021-11-28 03:00:00,, 2021-11-28 03:00:05,4.000,5.200 2021-11-28 03:00:10,1.600,7.600 2021-11-28 03:00:15,0.800,7.100 2021-11-28 03:00:20,16.600,8.400 2021-11-28 03:00:25,21.400,7.200 2021-11-28 03:00:30,21.200,6.800 2021-11-28 03:00:35,21.000,27.600 2021-11-28 03:00:40,12.400,33.800 2021-11-28 03:00:45,9.800,20.600