Chapter 5. Configuring the OpenTelemetry Collector for RHEL on public cloud platforms
Chapter 5. Configuring the OpenTelemetry Collector for RHEL on public cloud platforms When running RHEL on Amazon Web Services (AWS), you can use the OpenTelemetry (OTel) framework to maintain and debug your RHEL instances. RHEL includes the OTel Collector service, which you can use to manage logs. The OTel Collector gathers, processes, transforms, and exports logs to and from various formats and external back ends. You can also use the OTel Collector to aggregate the collected data and generate metrics useful for analytics services. 5.1. How the OpenTelemetry Collector works For RHEL on AWS, you can configure the OTel Collector service to receive, process, and export logs between the RHEL instance and the AWS telemetry analytics service to automatically manage telemetry data on your RHEL instance. The OTel Collector is a component of the OTel ecosystem, and has three stages in its workflow: a receiver, a processor, and an exporter. You can configure the workflow for any of these components in a YAML file based on your specific use case. Typically, the OTel Collector works as follows: A receiver collects telemetry data from data sources, such as applications and services. After the receiver ingest data, it passes to a processing phase, in which a chain of processors may be defined to transform the data. The exporter sends the telemetry data to the required destination. 5.2. Integration of OpenTelemetry with AWS CloudWatch Logs Integrating OTel with Amazon Web Services (AWS) for log management involves configuring the OTel Collector to use RHEL on AWS as exporter for logs. It works as follows: Configuring the exporter for the OTel Collector Enabling log connections Exporting data from the RHEL instance to AWS CloudWatch logs. As a result, you can gather log data from various sources at a single location to effectively manage log analysis. From the available features of AWS CloudWatch, RHEL instances currently support only logging. For details, see AWS Cloudwatch Logs exporter . 5.3. Configuring the OpenTelemetry Collector for journald logging To configure the OpenTelemetry (OTel) Collector, you need to modify the default configuration of the filelog receiver for capturing the journald service logs. This configuration involves defining the file path, log format, and parsing rules. With this setup, the collector processes and exports logs to services, such as AWS CloudWatch logs, to improve observability and metrics analysis of system components. Procedure Install the opentelemetry-collector package on a RHEL instance: Enable and start the service to transfer the logs from the RHEL instance to AWS CloudWatch Logs: To configure the OTel Collector to forward journald logs from the RHEL instance, create and edit the /etc/opentelemetry-collector/configs/10-cloudwatch-exporter.yaml file: ... exporters: awscloudwatchlogs: log_group_name: testing-logs-emf log_stream_name: testing-integrations-stream-emf raw_log: true region: us-east-1 endpoint: logs.us-east-1.amazonaws.com log_retention: 365 tags: sampleKey: sampleValue service: pipelines: logs: receivers: - journald exporters: - awscloudwatchlogs ... Restart the OTel Collector service: Create an IAM role for AWS CloudWatch agent from AWS console. For instructions, see Create IAM roles and users for use with the CloudWatch agent . Attach the role to the RHEL instance through AWS Console. For instructions, see Attach an IAM role to an instance . Restart the RHEL instance from AWS console to enable log exportation automatically. Optional: If you no longer want to export logs, stop logs transfer from the RHEL instance: Optional: If you no longer need this service, permanently disable logs transfer: Additional resources EMF logs Amazon CloudWatch Logs endpoints Tagging restrictions 5.4. Receivers for the OTel Collector Depending on the configuration, receivers gather telemetry based data such as logs and patterns of software use, from various devices and services at a single location for improved observability. Journald receiver The journald receiver in the OTel Collector captures logs from the journald service. This receiver accepts logs from system and application services, such as logs from the kernel, user, and applications, to provide improved observability. You can use journald logging for attributes like binary storage for faster indexing, user based permissions, and log size management. For details, see config option in Journald Receiver . 5.5. Processors for the OTel Collector Processors act as an intermediary between the receiver and the exporter and manipulate the data by, for example, adding, filtering, deleting, or transforming fields. Selection and order of processor depends on the signal type. Resource detection for AWS environment The resource detection processor collects a list of processors and detects information about the managed environment. It manages the details for telemetry data before exportation. For the snippet, see AWS EC2 configuration . 5.6. Exporters for the OTel Collector Exporters transmit processed data to specified devices or services, such as AWS CloudWatch Logs and the Debug exporter, based on the configuration and signal type. Exporters ensure compatibility with target services and facilitate integration with various systems. AWS Cloudwatch Logs exporter Note that, the given configuration currently supports only log type signals. Typically, it works as follows: Receiver sends logs to the OTel Collector. Processor processes logs in terms of modification or enhancement for exportation. The awscloudwatchlogs configuration sends processed telemetry to AWS CloudWatch Logs. For details, see: Example configuration for AWS CloudWatch Logs Exporter Details of configuration options In addition, the Collector provides extensions and processors to filter sensitive data, limit memory usage, and keep telemetry data on the disk for a certain period of time in case of a connection loss. Debug exporter The Debug Exporter prints traces and metrics to the standard output. Note that this exporter supports all signal types. You can modify the OTel Collector YAML configuration to include a console exporter, which will print the telemetry data to the console. Also, to make sure that journald captures the output, you can configure the receiver service if required. For details, see Debug exporter
dnf install -y opentelemetry-collector systemctl enable --now opentelemetry-collector.service exporters: awscloudwatchlogs: log_group_name: testing-logs-emf log_stream_name: testing-integrations-stream-emf raw_log: true region: us-east-1 endpoint: logs.us-east-1.amazonaws.com log_retention: 365 tags: sampleKey: sampleValue service: pipelines: logs: receivers: - journald exporters: - awscloudwatchlogs systemctl restart opentelemetry-collector.service systemctl stop opentelemetry-collector.service systemctl disable opentelemetry-collector.service