Chapter 3. Configuring the Collector
Chapter 3. Configuring the Collector 3.1. Configuring the Collector The Red Hat build of OpenTelemetry Operator uses a custom resource definition (CRD) file that defines the architecture and configuration settings to be used when creating and deploying the Red Hat build of OpenTelemetry resources. You can install the default configuration or modify the file. 3.1.1. OpenTelemetry Collector configuration options The OpenTelemetry Collector consists of five types of components that access telemetry data: Receivers Processors Exporters Connectors Extensions You can define multiple instances of components in a custom resource YAML file. When configured, these components must be enabled through pipelines defined in the spec.config.service section of the YAML file. As a best practice, only enable the components that you need. Example of the OpenTelemetry Collector custom resource file apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: cluster-collector namespace: tracing-system spec: mode: deployment observability: metrics: enableMetrics: true config: receivers: otlp: protocols: grpc: {} http: {} processors: {} exporters: otlp: endpoint: otel-collector-headless.tracing-system.svc:4317 tls: ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt" prometheus: endpoint: 0.0.0.0:8889 resource_to_telemetry_conversion: enabled: true # by default resource attributes are dropped service: 1 pipelines: traces: receivers: [otlp] processors: [] exporters: [otlp] metrics: receivers: [otlp] processors: [] exporters: [prometheus] 1 If a component is configured but not defined in the service section, the component is not enabled. Table 3.1. Parameters used by the Operator to define the OpenTelemetry Collector Parameter Description Values Default A receiver is how data gets into the Collector. By default, no receivers are configured. There must be at least one enabled receiver for a configuration to be considered valid. Receivers are enabled by being added to a pipeline. otlp , jaeger , prometheus , zipkin , kafka , opencensus None Processors run through the received data before it is exported. By default, no processors are enabled. batch , memory_limiter , resourcedetection , attributes , span , k8sattributes , filter , routing None An exporter sends data to one or more back ends or destinations. By default, no exporters are configured. There must be at least one enabled exporter for a configuration to be considered valid. Exporters are enabled by being added to a pipeline. Exporters might be used with their default settings, but many require configuration to specify at least the destination and security settings. otlp , otlphttp , debug , prometheus , kafka None Connectors join pairs of pipelines by consuming data as end-of-pipeline exporters and emitting data as start-of-pipeline receivers. Connectors can be used to summarize, replicate, or route consumed data. spanmetrics None Optional components for tasks that do not involve processing telemetry data. bearertokenauth , oauth2client , jaegerremotesampling , pprof , health_check , memory_ballast , zpages None Components are enabled by adding them to a pipeline under services.pipeline . You enable receivers for tracing by adding them under service.pipelines.traces . None You enable processors for tracing by adding them under service.pipelines.traces . None You enable exporters for tracing by adding them under service.pipelines.traces . None You enable receivers for metrics by adding them under service.pipelines.metrics . None You enable processors for metircs by adding them under service.pipelines.metrics . None You enable exporters for metrics by adding them under service.pipelines.metrics . None 3.1.2. Creating the required RBAC resources automatically Some Collector components require configuring the RBAC resources. Procedure Add the following permissions to the opentelemetry-operator-controller-manage service account so that the Red Hat build of OpenTelemetry Operator can create them automatically: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: generate-processors-rbac rules: - apiGroups: - rbac.authorization.k8s.io resources: - clusterrolebindings - clusterroles verbs: - create - delete - get - list - patch - update - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: generate-processors-rbac roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: generate-processors-rbac subjects: - kind: ServiceAccount name: opentelemetry-operator-controller-manager namespace: openshift-opentelemetry-operator 3.2. Receivers Receivers get data into the Collector. A receiver can be push or pull based. Generally, a receiver accepts data in a specified format, translates it into the internal format, and passes it to processors and exporters defined in the applicable pipelines. By default, no receivers are configured. One or more receivers must be configured. Receivers may support one or more data sources. Currently, the following General Availability and Technology Preview receivers are available for the Red Hat build of OpenTelemetry: OTLP Receiver Jaeger Receiver Host Metrics Receiver Kubernetes Objects Receiver Kubelet Stats Receiver Prometheus Receiver OTLP JSON File Receiver Zipkin Receiver Kafka Receiver Kubernetes Cluster Receiver OpenCensus Receiver Filelog Receiver Journald Receiver Kubernetes Events Receiver 3.2.1. OTLP Receiver The OTLP Receiver ingests traces, metrics, and logs by using the OpenTelemetry Protocol (OTLP). The OTLP Receiver ingests traces and metrics using the OpenTelemetry protocol (OTLP). OpenTelemetry Collector custom resource with an enabled OTLP Receiver # ... config: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem client_ca_file: client.pem 3 reload_interval: 1h 4 http: endpoint: 0.0.0.0:4318 5 tls: {} 6 service: pipelines: traces: receivers: [otlp] metrics: receivers: [otlp] # ... 1 The OTLP gRPC endpoint. If omitted, the default 0.0.0.0:4317 is used. 2 The server-side TLS configuration. Defines paths to TLS certificates. If omitted, the TLS is disabled. 3 The path to the TLS certificate at which the server verifies a client certificate. This sets the value of ClientCAs and ClientAuth to RequireAndVerifyClientCert in the TLSConfig . For more information, see the Config of the Golang TLS package . 4 Specifies the time interval at which the certificate is reloaded. If the value is not set, the certificate is never reloaded. The reload_interval field accepts a string containing valid units of time such as ns , us (or ms ), ms , s , m , h . 5 The OTLP HTTP endpoint. The default value is 0.0.0.0:4318 . 6 The server-side TLS configuration. For more information, see the grpc protocol configuration section. 3.2.2. Jaeger Receiver The Jaeger Receiver ingests traces in the Jaeger formats. OpenTelemetry Collector custom resource with an enabled Jaeger Receiver # ... config: receivers: jaeger: protocols: grpc: endpoint: 0.0.0.0:14250 1 thrift_http: endpoint: 0.0.0.0:14268 2 thrift_compact: endpoint: 0.0.0.0:6831 3 thrift_binary: endpoint: 0.0.0.0:6832 4 tls: {} 5 service: pipelines: traces: receivers: [jaeger] # ... 1 The Jaeger gRPC endpoint. If omitted, the default 0.0.0.0:14250 is used. 2 The Jaeger Thrift HTTP endpoint. If omitted, the default 0.0.0.0:14268 is used. 3 The Jaeger Thrift Compact endpoint. If omitted, the default 0.0.0.0:6831 is used. 4 The Jaeger Thrift Binary endpoint. If omitted, the default 0.0.0.0:6832 is used. 5 The server-side TLS configuration. See the OTLP Receiver configuration section for more details. 3.2.3. Host Metrics Receiver The Host Metrics Receiver ingests metrics in the OTLP format. OpenTelemetry Collector custom resource with an enabled Host Metrics Receiver apiVersion: v1 kind: ServiceAccount metadata: name: otel-hostfs-daemonset namespace: <namespace> # ... --- apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints allowHostDirVolumePlugin: true allowHostIPC: false allowHostNetwork: false allowHostPID: true allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: null defaultAddCapabilities: - SYS_ADMIN fsGroup: type: RunAsAny groups: [] metadata: name: otel-hostmetrics readOnlyRootFilesystem: true runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny supplementalGroups: type: RunAsAny users: - system:serviceaccount:<namespace>:otel-hostfs-daemonset volumes: - configMap - emptyDir - hostPath - projected # ... --- apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otel namespace: <namespace> spec: serviceAccount: otel-hostfs-daemonset mode: daemonset volumeMounts: - mountPath: /hostfs name: host readOnly: true volumes: - hostPath: path: / name: host config: receivers: hostmetrics: collection_interval: 10s 1 initial_delay: 1s 2 root_path: / 3 scrapers: 4 cpu: {} memory: {} disk: {} service: pipelines: metrics: receivers: [hostmetrics] # ... 1 Sets the time interval for host metrics collection. If omitted, the default value is 1m . 2 Sets the initial time delay for host metrics collection. If omitted, the default value is 1s . 3 Configures the root_path so that the Host Metrics Receiver knows where the root filesystem is. If running multiple instances of the Host Metrics Receiver, set the same root_path value for each instance. 4 Lists the enabled host metrics scrapers. Available scrapers are cpu , disk , load , filesystem , memory , network , paging , processes , and process . 3.2.4. Kubernetes Objects Receiver The Kubernetes Objects Receiver pulls or watches objects to be collected from the Kubernetes API server. This receiver watches primarily Kubernetes events, but it can collect any type of Kubernetes objects. This receiver gathers telemetry for the cluster as a whole, so only one instance of this receiver suffices for collecting all the data. Important The Kubernetes Objects Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with an enabled Kubernetes Objects Receiver apiVersion: v1 kind: ServiceAccount metadata: name: otel-k8sobj namespace: <namespace> # ... --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-k8sobj namespace: <namespace> rules: - apiGroups: - "" resources: - events - pods verbs: - get - list - watch - apiGroups: - "events.k8s.io" resources: - events verbs: - watch - list # ... --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-k8sobj subjects: - kind: ServiceAccount name: otel-k8sobj namespace: <namespace> roleRef: kind: ClusterRole name: otel-k8sobj apiGroup: rbac.authorization.k8s.io # ... --- apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otel-k8s-obj namespace: <namespace> spec: serviceAccount: otel-k8sobj mode: deployment config: receivers: k8sobjects: auth_type: serviceAccount objects: - name: pods 1 mode: pull 2 interval: 30s 3 label_selector: 4 field_selector: 5 namespaces: [<namespace>,...] 6 - name: events mode: watch exporters: debug: service: pipelines: logs: receivers: [k8sobjects] exporters: [debug] # ... 1 The Resource name that this receiver observes: for example, pods , deployments , or events . 2 The observation mode that this receiver uses: pull or watch . 3 Only applicable to the pull mode. The request interval for pulling an object. If omitted, the default value is 1h . 4 The label selector to define targets. 5 The field selector to filter targets. 6 The list of namespaces to collect events from. If omitted, the default value is all . 3.2.5. Kubelet Stats Receiver The Kubelet Stats Receiver extracts metrics related to nodes, pods, containers, and volumes from the kubelet's API server. These metrics are then channeled through the metrics-processing pipeline for additional analysis. OpenTelemetry Collector custom resource with an enabled Kubelet Stats Receiver # ... config: receivers: kubeletstats: collection_interval: 20s auth_type: "serviceAccount" endpoint: "https://USD{env:K8S_NODE_NAME}:10250" insecure_skip_verify: true service: pipelines: metrics: receivers: [kubeletstats] env: - name: K8S_NODE_NAME 1 valueFrom: fieldRef: fieldPath: spec.nodeName # ... 1 Sets the K8S_NODE_NAME to authenticate to the API. The Kubelet Stats Receiver requires additional permissions for the service account used for running the OpenTelemetry Collector. Permissions required by the service account apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: [''] resources: ['nodes/stats'] verbs: ['get', 'watch', 'list'] - apiGroups: [""] resources: ["nodes/proxy"] 1 verbs: ["get"] # ... 1 The permissions required when using the extra_metadata_labels or request_utilization or limit_utilization metrics. 3.2.6. Prometheus Receiver The Prometheus Receiver scrapes the metrics endpoints. Important The Prometheus Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with an enabled Prometheus Receiver # ... config: receivers: prometheus: config: scrape_configs: 1 - job_name: 'my-app' 2 scrape_interval: 5s 3 static_configs: - targets: ['my-app.example.svc.cluster.local:8888'] 4 service: pipelines: metrics: receivers: [prometheus] # ... 1 Scrapes configurations using the Prometheus format. 2 The Prometheus job name. 3 The lnterval for scraping the metrics data. Accepts time units. The default value is 1m . 4 The targets at which the metrics are exposed. This example scrapes the metrics from a my-app application in the example project. 3.2.7. OTLP JSON File Receiver The OTLP JSON File Receiver extracts pipeline information from files containing data in the ProtoJSON format and conforming to the OpenTelemetry Protocol specification. The receiver watches a specified directory for changes such as created or modified files to process. Important The OTLP JSON File Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled OTLP JSON File Receiver # ... config: otlpjsonfile: include: - "/var/log/*.log" 1 exclude: - "/var/log/test.log" 2 # ... 1 The list of file path glob patterns to watch. 2 The list of file path glob patterns to ignore. 3.2.8. Zipkin Receiver The Zipkin Receiver ingests traces in the Zipkin v1 and v2 formats. OpenTelemetry Collector custom resource with the enabled Zipkin Receiver # ... config: receivers: zipkin: endpoint: 0.0.0.0:9411 1 tls: {} 2 service: pipelines: traces: receivers: [zipkin] # ... 1 The Zipkin HTTP endpoint. If omitted, the default 0.0.0.0:9411 is used. 2 The server-side TLS configuration. See the OTLP Receiver configuration section for more details. 3.2.9. Kafka Receiver The Kafka Receiver receives traces, metrics, and logs from Kafka in the OTLP format. Important The Kafka Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled Kafka Receiver # ... config: receivers: kafka: brokers: ["localhost:9092"] 1 protocol_version: 2.0.0 2 topic: otlp_spans 3 auth: plain_text: 4 username: example password: example tls: 5 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 6 server_name_override: kafka.example.corp 7 service: pipelines: traces: receivers: [kafka] # ... 1 The list of Kafka brokers. The default is localhost:9092 . 2 The Kafka protocol version. For example, 2.0.0 . This is a required field. 3 The name of the Kafka topic to read from. The default is otlp_spans . 4 The plain text authentication configuration. If omitted, plain text authentication is disabled. 5 The client-side TLS configuration. Defines paths to the TLS certificates. If omitted, TLS authentication is disabled. 6 Disables verifying the server's certificate chain and host name. The default is false . 7 ServerName indicates the name of the server requested by the client to support virtual hosting. 3.2.10. Kubernetes Cluster Receiver The Kubernetes Cluster Receiver gathers cluster metrics and entity events from the Kubernetes API server. It uses the Kubernetes API to receive information about updates. Authentication for this receiver is only supported through service accounts. Important The Kubernetes Cluster Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled Kubernetes Cluster Receiver # ... config: receivers: k8s_cluster: distribution: openshift collection_interval: 10s exporters: debug: {} service: pipelines: metrics: receivers: [k8s_cluster] exporters: [debug] logs/entity_events: receivers: [k8s_cluster] exporters: [debug] # ... This receiver requires a configured service account, RBAC rules for the cluster role, and the cluster role binding that binds the RBAC with the service account. ServiceAccount object apiVersion: v1 kind: ServiceAccount metadata: labels: app: otelcontribcol name: otelcontribcol # ... RBAC rules for the ClusterRole object apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otelcontribcol labels: app: otelcontribcol rules: - apiGroups: - quota.openshift.io resources: - clusterresourcequotas verbs: - get - list - watch - apiGroups: - "" resources: - events - namespaces - namespaces/status - nodes - nodes/spec - pods - pods/status - replicationcontrollers - replicationcontrollers/status - resourcequotas - services verbs: - get - list - watch - apiGroups: - apps resources: - daemonsets - deployments - replicasets - statefulsets verbs: - get - list - watch - apiGroups: - extensions resources: - daemonsets - deployments - replicasets verbs: - get - list - watch - apiGroups: - batch resources: - jobs - cronjobs verbs: - get - list - watch - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - get - list - watch # ... ClusterRoleBinding object apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otelcontribcol labels: app: otelcontribcol roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: otelcontribcol subjects: - kind: ServiceAccount name: otelcontribcol namespace: default # ... 3.2.11. OpenCensus Receiver The OpenCensus Receiver provides backwards compatibility with the OpenCensus project for easier migration of instrumented codebases. It receives metrics and traces in the OpenCensus format via gRPC or HTTP and Json. OpenTelemetry Collector custom resource with the enabled OpenCensus Receiver # ... config: receivers: opencensus: endpoint: 0.0.0.0:9411 1 tls: 2 cors_allowed_origins: 3 - https://*.<example>.com service: pipelines: traces: receivers: [opencensus] # ... 1 The OpenCensus endpoint. If omitted, the default is 0.0.0.0:55678 . 2 The server-side TLS configuration. See the OTLP Receiver configuration section for more details. 3 You can also use the HTTP JSON endpoint to optionally configure CORS, which is enabled by specifying a list of allowed CORS origins in this field. Wildcards with * are accepted under the cors_allowed_origins . To match any origin, enter only * . 3.2.12. Filelog Receiver The Filelog Receiver tails and parses logs from files. Important The Filelog Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled Filelog Receiver that tails a text file # ... config: receivers: filelog: include: [ /simple.log ] 1 operators: 2 - type: regex_parser regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)USD' timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' severity: parse_from: attributes.sev # ... 1 A list of file glob patterns that match the file paths to be read. 2 An array of Operators. Each Operator performs a simple task such as parsing a timestamp or JSON. To process logs into a desired format, chain the Operators together. 3.2.13. Journald Receiver The Journald Receiver parses journald events from the systemd journal and sends them as logs. Important The Journald Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled Journald Receiver apiVersion: v1 kind: Namespace metadata: name: otel-journald labels: security.openshift.io/scc.podSecurityLabelSync: "false" pod-security.kubernetes.io/enforce: "privileged" pod-security.kubernetes.io/audit: "privileged" pod-security.kubernetes.io/warn: "privileged" # ... --- apiVersion: v1 kind: ServiceAccount metadata: name: privileged-sa namespace: otel-journald # ... --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-journald-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:openshift:scc:privileged subjects: - kind: ServiceAccount name: privileged-sa namespace: otel-journald # ... --- apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otel-journald-logs namespace: otel-journald spec: mode: daemonset serviceAccount: privileged-sa securityContext: allowPrivilegeEscalation: false capabilities: drop: - CHOWN - DAC_OVERRIDE - FOWNER - FSETID - KILL - NET_BIND_SERVICE - SETGID - SETPCAP - SETUID readOnlyRootFilesystem: true seLinuxOptions: type: spc_t seccompProfile: type: RuntimeDefault config: receivers: journald: files: /var/log/journal/*/* priority: info 1 units: 2 - kubelet - crio - init.scope - dnsmasq all: true 3 retry_on_failure: enabled: true 4 initial_interval: 1s 5 max_interval: 30s 6 max_elapsed_time: 5m 7 processors: exporters: debug: {} service: pipelines: logs: receivers: [journald] exporters: [debug] volumeMounts: - name: journal-logs mountPath: /var/log/journal/ readOnly: true volumes: - name: journal-logs hostPath: path: /var/log/journal tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule # ... 1 Filters output by message priorities or priority ranges. The default value is info . 2 Lists the units to read entries from. If empty, entries are read from all units. 3 Includes very long logs and logs with unprintable characters. The default value is false . 4 If set to true , the receiver pauses reading a file and attempts to resend the current batch of logs when encountering an error from downstream components. The default value is false . 5 The time interval to wait after the first failure before retrying. The default value is 1s . The units are ms , s , m , h . 6 The upper bound for the retry backoff interval. When this value is reached, the time interval between consecutive retry attempts remains constant at this value. The default value is 30s . The supported units are ms , s , m , h . 7 The maximum time interval, including retry attempts, for attempting to send a logs batch to a downstream consumer. When this value is reached, the data are discarded. If the set value is 0 , retrying never stops. The default value is 5m . The supported units are ms , s , m , h . 3.2.14. Kubernetes Events Receiver The Kubernetes Events Receiver collects events from the Kubernetes API server. The collected events are converted into logs. Important The Kubernetes Events Receiver is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenShift Container Platform permissions required for the Kubernetes Events Receiver apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-collector labels: app: otel-collector rules: - apiGroups: - "" resources: - events - namespaces - namespaces/status - nodes - nodes/spec - pods - pods/status - replicationcontrollers - replicationcontrollers/status - resourcequotas - services verbs: - get - list - watch - apiGroups: - apps resources: - daemonsets - deployments - replicasets - statefulsets verbs: - get - list - watch - apiGroups: - extensions resources: - daemonsets - deployments - replicasets verbs: - get - list - watch - apiGroups: - batch resources: - jobs - cronjobs verbs: - get - list - watch - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - get - list - watch # ... OpenTelemetry Collector custom resource with the enabled Kubernetes Event Receiver # ... serviceAccount: otel-collector 1 config: receivers: k8s_events: namespaces: [project1, project2] 2 service: pipelines: logs: receivers: [k8s_events] # ... 1 The service account of the Collector that has the required ClusterRole otel-collector RBAC. 2 The list of namespaces to collect events from. The default value is empty, which means that all namespaces are collected. 3.2.15. Additional resources OpenTelemetry Protocol (OTLP) documentation 3.3. Processors Processors process the data between it is received and exported. Processors are optional. By default, no processors are enabled. Processors must be enabled for every data source. Not all processors support all data sources. Depending on the data source, multiple processors might be enabled. Note that the order of processors matters. Currently, the following General Availability and Technology Preview processors are available for the Red Hat build of OpenTelemetry: Batch Processor Memory Limiter Processor Resource Detection Processor Attributes Processor Resource Processor Span Processor Kubernetes Attributes Processor Filter Processor Routing Processor Cumulative-to-Delta Processor Group-by-Attributes Processor Transform Processor 3.3.1. Batch Processor The Batch Processor batches traces and metrics to reduce the number of outgoing connections needed to transfer the telemetry information. Example of the OpenTelemetry Collector custom resource when using the Batch Processor # ... config: processors: batch: timeout: 5s send_batch_max_size: 10000 service: pipelines: traces: processors: [batch] metrics: processors: [batch] # ... Table 3.2. Parameters used by the Batch Processor Parameter Description Default timeout Sends the batch after a specific time duration and irrespective of the batch size. 200ms send_batch_size Sends the batch of telemetry data after the specified number of spans or metrics. 8192 send_batch_max_size The maximum allowable size of the batch. Must be equal or greater than the send_batch_size . 0 metadata_keys When activated, a batcher instance is created for each unique set of values found in the client.Metadata . [] metadata_cardinality_limit When the metadata_keys are populated, this configuration restricts the number of distinct metadata key-value combinations processed throughout the duration of the process. 1000 3.3.2. Memory Limiter Processor The Memory Limiter Processor periodically checks the Collector's memory usage and pauses data processing when the soft memory limit is reached. This processor supports traces, metrics, and logs. The preceding component, which is typically a receiver, is expected to retry sending the same data and may apply a backpressure to the incoming data. When memory usage exceeds the hard limit, the Memory Limiter Processor forces garbage collection to run. Example of the OpenTelemetry Collector custom resource when using the Memory Limiter Processor # ... config: processors: memory_limiter: check_interval: 1s limit_mib: 4000 spike_limit_mib: 800 service: pipelines: traces: processors: [batch] metrics: processors: [batch] # ... Table 3.3. Parameters used by the Memory Limiter Processor Parameter Description Default check_interval Time between memory usage measurements. The optimal value is 1s . For spiky traffic patterns, you can decrease the check_interval or increase the spike_limit_mib . 0s limit_mib The hard limit, which is the maximum amount of memory in MiB allocated on the heap. Typically, the total memory usage of the OpenTelemetry Collector is about 50 MiB greater than this value. 0 spike_limit_mib Spike limit, which is the maximum expected spike of memory usage in MiB. The optimal value is approximately 20% of limit_mib . To calculate the soft limit, subtract the spike_limit_mib from the limit_mib . 20% of limit_mib limit_percentage Same as the limit_mib but expressed as a percentage of the total available memory. The limit_mib setting takes precedence over this setting. 0 spike_limit_percentage Same as the spike_limit_mib but expressed as a percentage of the total available memory. Intended to be used with the limit_percentage setting. 0 3.3.3. Resource Detection Processor The Resource Detection Processor identifies host resource details in alignment with OpenTelemetry's resource semantic standards. Using the detected information, this processor can add or replace the resource values in telemetry data. This processor supports traces and metrics. You can use this processor with multiple detectors such as the Docket metadata detector or the OTEL_RESOURCE_ATTRIBUTES environment variable detector. Important The Resource Detection Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenShift Container Platform permissions required for the Resource Detection Processor kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: ["config.openshift.io"] resources: ["infrastructures", "infrastructures/status"] verbs: ["get", "watch", "list"] # ... OpenTelemetry Collector using the Resource Detection Processor # ... config: processors: resourcedetection: detectors: [openshift] override: true service: pipelines: traces: processors: [resourcedetection] metrics: processors: [resourcedetection] # ... OpenTelemetry Collector using the Resource Detection Processor with an environment variable detector # ... config: processors: resourcedetection/env: detectors: [env] 1 timeout: 2s override: false # ... 1 Specifies which detector to use. In this example, the environment detector is specified. 3.3.4. Attributes Processor The Attributes Processor can modify attributes of a span, log, or metric. You can configure this processor to filter and match input data and include or exclude such data for specific actions. Important The Attributes Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . This processor operates on a list of actions, executing them in the order specified in the configuration. The following actions are supported: Insert Inserts a new attribute into the input data when the specified key does not already exist. Update Updates an attribute in the input data if the key already exists. Upsert Combines the insert and update actions: Inserts a new attribute if the key does not exist yet. Updates the attribute if the key already exists. Delete Removes an attribute from the input data. Hash Hashes an existing attribute value as SHA1. Extract Extracts values by using a regular expression rule from the input key to the target keys defined in the rule. If a target key already exists, it is overridden similarly to the Span Processor's to_attributes setting with the existing attribute as the source. Convert Converts an existing attribute to a specified type. OpenTelemetry Collector using the Attributes Processor # ... config: processors: attributes/example: actions: - key: db.table action: delete - key: redacted_span value: true action: upsert - key: copy_key from_attribute: key_original action: update - key: account_id value: 2245 action: insert - key: account_password action: delete - key: account_email action: hash - key: http.status_code action: convert converted_type: int # ... 3.3.5. Resource Processor The Resource Processor applies changes to the resource attributes. This processor supports traces, metrics, and logs. Important The Resource Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector using the Resource Detection Processor # ... config: processors: attributes: - key: cloud.availability_zone value: "zone-1" action: upsert - key: k8s.cluster.name from_attribute: k8s-cluster action: insert - key: redundant-attribute action: delete # ... Attributes represent the actions that are applied to the resource attributes, such as delete the attribute, insert the attribute, or upsert the attribute. 3.3.6. Span Processor The Span Processor modifies the span name based on its attributes or extracts the span attributes from the span name. This processor can also change the span status and include or exclude spans. This processor supports traces. Span renaming requires specifying attributes for the new name by using the from_attributes configuration. Important The Span Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector using the Span Processor for renaming a span # ... config: processors: span: name: from_attributes: [<key1>, <key2>, ...] 1 separator: <value> 2 # ... 1 Defines the keys to form the new span name. 2 An optional separator. You can use this processor to extract attributes from the span name. OpenTelemetry Collector using the Span Processor for extracting attributes from a span name # ... config: processors: span/to_attributes: name: to_attributes: rules: - ^\/api\/v1\/document\/(?P<documentId>.*)\/updateUSD 1 # ... 1 This rule defines how the extraction is to be executed. You can define more rules: for example, in this case, if the regular expression matches the name, a documentID attibute is created. In this example, if the input span name is /api/v1/document/12345678/update , this results in the /api/v1/document/{documentId}/update output span name, and a new "documentId"="12345678" attribute is added to the span. You can have the span status modified. OpenTelemetry Collector using the Span Processor for status change # ... config: processors: span/set_status: status: code: Error description: "<error_description>" # ... 3.3.7. Kubernetes Attributes Processor The Kubernetes Attributes Processor enables automatic configuration of spans, metrics, and log resource attributes by using the Kubernetes metadata. This processor supports traces, metrics, and logs. This processor automatically identifies the Kubernetes resources, extracts the metadata from them, and incorporates this extracted metadata as resource attributes into relevant spans, metrics, and logs. It utilizes the Kubernetes API to discover all pods operating within a cluster, maintaining records of their IP addresses, pod UIDs, and other relevant metadata. Minimum OpenShift Container Platform permissions required for the Kubernetes Attributes Processor kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: [''] resources: ['pods', 'namespaces'] verbs: ['get', 'watch', 'list'] # ... OpenTelemetry Collector using the Kubernetes Attributes Processor # ... config: processors: k8sattributes: filter: node_from_env_var: KUBE_NODE_NAME # ... 3.3.8. Filter Processor The Filter Processor leverages the OpenTelemetry Transformation Language to establish criteria for discarding telemetry data. If any of these conditions are satisfied, the telemetry data are discarded. You can combine the conditions by using the logical OR operator. This processor supports traces, metrics, and logs. Important The Filter Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with an enabled OTLP Exporter # ... config: processors: filter/ottl: error_mode: ignore 1 traces: span: - 'attributes["container.name"] == "app_container_1"' 2 - 'resource.attributes["host.name"] == "localhost"' 3 # ... 1 Defines the error mode. When set to ignore , ignores errors returned by conditions. When set to propagate , returns the error up the pipeline. An error causes the payload to be dropped from the Collector. 2 Filters the spans that have the container.name == app_container_1 attribute. 3 Filters the spans that have the host.name == localhost resource attribute. 3.3.9. Routing Processor The Routing Processor routes logs, metrics, or traces to specific exporters. This processor can read a header from an incoming gRPC or plain HTTP request or read a resource attribute, and then direct the trace information to relevant exporters according to the read value. Important The Routing Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with an enabled OTLP Exporter # ... config: processors: routing: from_attribute: X-Tenant 1 default_exporters: 2 - jaeger table: 3 - value: acme exporters: [jaeger/acme] exporters: jaeger: endpoint: localhost:14250 jaeger/acme: endpoint: localhost:24250 # ... 1 The HTTP header name for the lookup value when performing the route. 2 The default exporter when the attribute value is not present in the table in the section. 3 The table that defines which values are to be routed to which exporters. Optionally, you can create an attribute_source configuration, which defines where to look for the attribute that you specify in the from_attribute field. The supported values are context for searching the context including the HTTP headers, and resource for searching the resource attributes. 3.3.10. Cumulative-to-Delta Processor The Cumulative-to-Delta Processor processor converts monotonic, cumulative-sum, and histogram metrics to monotonic delta metrics. You can filter metrics by using the include: or exclude: fields and specifying the strict or regexp metric name matching. This processor does not convert non-monotonic sums and exponential histograms. Important The Cumulative-to-Delta Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Example of an OpenTelemetry Collector custom resource with an enabled Cumulative-to-Delta Processor # ... config: processors: cumulativetodelta: include: 1 match_type: strict 2 metrics: 3 - <metric_1_name> - <metric_2_name> exclude: 4 match_type: regexp metrics: - "<regular_expression_for_metric_names>" # ... 1 Optional: Configures which metrics to include. When omitted, all metrics, except for those listed in the exclude field, are converted to delta metrics. 2 Defines a value provided in the metrics field as a strict exact match or regexp regular expression. 3 Lists the metric names, which are exact matches or matches for regular expressions, of the metrics to be converted to delta metrics. If a metric matches both the include and exclude filters, the exclude filter takes precedence. 4 Optional: Configures which metrics to exclude. When omitted, no metrics are excluded from conversion to delta metrics. 3.3.11. Group-by-Attributes Processor The Group-by-Attributes Processor groups all spans, log records, and metric datapoints that share the same attributes by reassigning them to a Resource that matches those attributes. Important The Group-by-Attributes Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . At minimum, configuring this processor involves specifying an array of attribute keys to be used to group spans, log records, or metric datapoints together, as in the following example: # ... config: processors: groupbyattrs: keys: 1 - <key1> 2 - <key2> # ... 1 Specifies attribute keys to group by. 2 If a processed span, log record, or metric datapoint contains at least one of the specified attribute keys, it is reassigned to a Resource that shares the same attribute values; and if no such Resource exists, a new one is created. If none of the specified attribute keys is present in the processed span, log record, or metric datapoint, then it remains associated with its current Resource. Multiple instances of the same Resource are consolidated. 3.3.12. Transform Processor The Transform Processor enables modification of telemetry data according to specified rules and in the OpenTelemetry Transformation Language (OTTL) . For each signal type, the processor processes a series of conditions and statements associated with a specific OTTL Context type and then executes them in sequence on incoming telemetry data as specified in the configuration. Each condition and statement can access and modify telemetry data by using various functions, allowing conditions to dictate if a function is to be executed. All statements are written in the OTTL. You can configure multiple context statements for different signals, traces, metrics, and logs. The value of the context type specifies which OTTL Context the processor must use when interpreting the associated statements. Important The Transform Processor is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Configuration summary # ... config: processors: transform: error_mode: ignore 1 <trace|metric|log>_statements: 2 - context: <string> 3 conditions: 4 - <string> - <string> statements: 5 - <string> - <string> - <string> - context: <string> statements: - <string> - <string> - <string> # ... 1 Optional: See the following table "Values for the optional error_mode field". 2 Indicates a signal to be transformed. 3 See the following table "Values for the context field". 4 Optional: Conditions for performing a transformation. Configuration example # ... config: transform: error_mode: ignore trace_statements: 1 - context: resource statements: - keep_keys(attributes, ["service.name", "service.namespace", "cloud.region", "process.command_line"]) 2 - replace_pattern(attributes["process.command_line"], "password\\=[^\\s]*(\\s?)", "password=***") 3 - limit(attributes, 100, []) - truncate_all(attributes, 4096) - context: span 4 statements: - set(status.code, 1) where attributes["http.path"] == "/health" - set(name, attributes["http.route"]) - replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}") - limit(attributes, 100, []) - truncate_all(attributes, 4096) # ... 1 Transforms a trace signal. 2 Keeps keys on the resources. 3 Replaces attributes and replaces string characters in password fields with asterisks. 4 Performs transformations at the span level. Table 3.4. Values for the context field Signal Statement Valid Contexts trace_statements resource , scope , span , spanevent metric_statements resource , scope , metric , datapoint log_statements resource , scope , log Table 3.5. Values for the optional error_mode field Value Description ignore Ignores and logs errors returned by statements and then continues to the statement. silent Ignores and doesn't log errors returned by statements and then continues to the statement. propagate Returns errors up the pipeline and drops the payload. Implicit default. 3.3.13. Additional resources OpenTelemetry Protocol (OTLP) documentation 3.4. Exporters Exporters send data to one or more back ends or destinations. An exporter can be push or pull based. By default, no exporters are configured. One or more exporters must be configured. Exporters can support one or more data sources. Exporters might be used with their default settings, but many exporters require configuration to specify at least the destination and security settings. Currently, the following General Availability and Technology Preview exporters are available for the Red Hat build of OpenTelemetry: OTLP Exporter OTLP HTTP Exporter Debug Exporter Load Balancing Exporter Prometheus Exporter Prometheus Remote Write Exporter Kafka Exporter AWS CloudWatch Exporter AWS EMF Exporter AWS X-Ray Exporter File Exporter 3.4.1. OTLP Exporter The OTLP gRPC Exporter exports traces and metrics by using the OpenTelemetry protocol (OTLP). OpenTelemetry Collector custom resource with the enabled OTLP Exporter # ... config: exporters: otlp: endpoint: tempo-ingester:4317 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 3 insecure_skip_verify: false # 4 reload_interval: 1h 5 server_name_override: <name> 6 headers: 7 X-Scope-OrgID: "dev" service: pipelines: traces: exporters: [otlp] metrics: exporters: [otlp] # ... 1 The OTLP gRPC endpoint. If the https:// scheme is used, then client transport security is enabled and overrides the insecure setting in the tls . 2 The client-side TLS configuration. Defines paths to TLS certificates. 3 Disables client transport security when set to true . The default value is false by default. 4 Skips verifying the certificate when set to true . The default value is false . 5 Specifies the time interval at which the certificate is reloaded. If the value is not set, the certificate is never reloaded. The reload_interval accepts a string containing valid units of time such as ns , us (or ms ), ms , s , m , h . 6 Overrides the virtual host name of authority such as the authority header field in requests. You can use this for testing. 7 Headers are sent for every request performed during an established connection. 3.4.2. OTLP HTTP Exporter The OTLP HTTP Exporter exports traces and metrics by using the OpenTelemetry protocol (OTLP). OpenTelemetry Collector custom resource with the enabled OTLP Exporter # ... config: exporters: otlphttp: endpoint: http://tempo-ingester:4318 1 tls: 2 headers: 3 X-Scope-OrgID: "dev" disable_keep_alives: false 4 service: pipelines: traces: exporters: [otlphttp] metrics: exporters: [otlphttp] # ... 1 The OTLP HTTP endpoint. If the https:// scheme is used, then client transport security is enabled and overrides the insecure setting in the tls . 2 The client side TLS configuration. Defines paths to TLS certificates. 3 Headers are sent in every HTTP request. 4 If true, disables HTTP keep-alives. It will only use the connection to the server for a single HTTP request. 3.4.3. Debug Exporter The Debug Exporter prints traces and metrics to the standard output. OpenTelemetry Collector custom resource with the enabled Debug Exporter # ... config: exporters: debug: verbosity: detailed 1 sampling_initial: 5 2 sampling_thereafter: 200 3 use_internal_logger: true 4 service: pipelines: traces: exporters: [debug] metrics: exporters: [debug] # ... 1 Verbosity of the debug export: detailed , normal , or basic . When set to detailed , pipeline data are verbosely logged. Defaults to normal . 2 Initial number of messages logged per second. The default value is 2 messages per second. 3 Sampling rate after the initial number of messages, the value in sampling_initial , has been logged. Disabled by default with the default 1 value. Sampling is enabled with values greater than 1 . For more information, see the page for the sampler function in the zapcore package on the Go Project's website. 4 When set to true , enables output from the Collector's internal logger for the exporter. 3.4.4. Load Balancing Exporter The Load Balancing Exporter consistently exports spans, metrics, and logs according to the routing_key configuration. Important The Load Balancing Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled Load Balancing Exporter # ... config: exporters: loadbalancing: routing_key: "service" 1 protocol: otlp: 2 timeout: 1s resolver: 3 static: 4 hostnames: - backend-1:4317 - backend-2:4317 dns: 5 hostname: otelcol-headless.observability.svc.cluster.local k8s: 6 service: lb-svc.kube-public ports: - 15317 - 16317 # ... 1 The routing_key: service exports spans for the same service name to the same Collector instance to provide accurate aggregation. The routing_key: traceID exports spans based on their traceID . The implicit default is traceID based routing. 2 The OTLP is the only supported load-balancing protocol. All options of the OTLP exporter are supported. 3 You can configure only one resolver. 4 The static resolver distributes the load across the listed endpoints. 5 You can use the DNS resolver only with a Kubernetes headless service. 6 The Kubernetes resolver is recommended. 3.4.5. Prometheus Exporter The Prometheus Exporter exports metrics in the Prometheus or OpenMetrics formats. Important The Prometheus Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled Prometheus Exporter # ... config: exporters: prometheus: endpoint: 0.0.0.0:8889 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem namespace: prefix 3 const_labels: 4 label1: value1 enable_open_metrics: true 5 resource_to_telemetry_conversion: 6 enabled: true metric_expiration: 180m 7 add_metric_suffixes: false 8 service: pipelines: metrics: exporters: [prometheus] # ... 1 The network endpoint where the metrics are exposed. The Red Hat build of OpenTelemetry Operator automatically exposes the port specified in the endpoint field to the <instance_name>-collector service. 2 The server-side TLS configuration. Defines paths to TLS certificates. 3 If set, exports metrics under the provided value. 4 Key-value pair labels that are applied for every exported metric. 5 If true , metrics are exported by using the OpenMetrics format. Exemplars are only exported in the OpenMetrics format and only for histogram and monotonic sum metrics such as counter . Disabled by default. 6 If enabled is true , all the resource attributes are converted to metric labels. Disabled by default. 7 Defines how long metrics are exposed without updates. The default is 5m . 8 Adds the metrics types and units suffixes. Must be disabled if the monitor tab in the Jaeger console is enabled. The default is true . Note When the spec.observability.metrics.enableMetrics field in the OpenTelemetryCollector custom resource (CR) is set to true , the OpenTelemetryCollector CR automatically creates a Prometheus ServiceMonitor or PodMonitor CR to enable Prometheus to scrape your metrics. 3.4.6. Prometheus Remote Write Exporter The Prometheus Remote Write Exporter exports metrics to compatible back ends. Important The Prometheus Remote Write Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled Prometheus Remote Write Exporter # ... config: exporters: prometheusremotewrite: endpoint: "https://my-prometheus:7900/api/v1/push" 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem target_info: true 3 export_created_metric: true 4 max_batch_size_bytes: 3000000 5 service: pipelines: metrics: exporters: [prometheusremotewrite] # ... 1 Endpoint for sending the metrics. 2 Server-side TLS configuration. Defines paths to TLS certificates. 3 When set to true , creates a target_info metric for each resource metric. 4 When set to true , exports a _created metric for the Summary, Histogram, and Monotonic Sum metric points. 5 Maximum size of the batch of samples that is sent to the remote write endpoint. Exceeding this value results in batch splitting. The default value is 3000000 , which is approximately 2.861 megabytes. Warning This exporter drops non-cumulative monotonic, histogram, and summary OTLP metrics. You must enable the --web.enable-remote-write-receiver feature flag on the remote Prometheus instance. Without it, pushing the metrics to the instance using this exporter fails. 3.4.7. Kafka Exporter The Kafka Exporter exports logs, metrics, and traces to Kafka. This exporter uses a synchronous producer that blocks and does not batch messages. You must use it with batch and queued retry processors for higher throughput and resiliency. Important The Kafka Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled Kafka Exporter # ... config: exporters: kafka: brokers: ["localhost:9092"] 1 protocol_version: 2.0.0 2 topic: otlp_spans 3 auth: plain_text: 4 username: example password: example tls: 5 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 6 server_name_override: kafka.example.corp 7 service: pipelines: traces: exporters: [kafka] # ... 1 The list of Kafka brokers. The default is localhost:9092 . 2 The Kafka protocol version. For example, 2.0.0 . This is a required field. 3 The name of the Kafka topic to read from. The following are the defaults: otlp_spans for traces, otlp_metrics for metrics, otlp_logs for logs. 4 The plain text authentication configuration. If omitted, plain text authentication is disabled. 5 The client-side TLS configuration. Defines paths to the TLS certificates. If omitted, TLS authentication is disabled. 6 Disables verifying the server's certificate chain and host name. The default is false . 7 ServerName indicates the name of the server requested by the client to support virtual hosting. 3.4.8. AWS CloudWatch Logs Exporter The AWS CloudWatch Logs Exporter sends logs data to the Amazon CloudWatch Logs service and signs requests by using the AWS SDK for Go and the default credential provider chain. Important The AWS CloudWatch Logs Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled AWS CloudWatch Logs Exporter # ... config: exporters: awscloudwatchlogs: log_group_name: "<group_name_of_amazon_cloudwatch_logs>" 1 log_stream_name: "<log_stream_of_amazon_cloudwatch_logs>" 2 region: <aws_region_of_log_stream> 3 endpoint: <service_endpoint_of_amazon_cloudwatch_logs> 4 log_retention: <supported_value_in_days> 5 # ... 1 Required. If the log group does not exist yet, it is automatically created. 2 Required. If the log stream does not exist yet, it is automatically created. 3 Optional. If the AWS region is not already set in the default credential chain, you must specify it. 4 Optional. You can override the default Amazon CloudWatch Logs service endpoint to which the requests are forwarded. For the list of service endpoints by region, see Amazon CloudWatch Logs endpoints and quotas (AWS General Reference). 5 Optional. With this parameter, you can set the log retention policy for new Amazon CloudWatch log groups. If this parameter is omitted or set to 0 , the logs never expire by default. Supported values for retention in days are 1 , 3 , 5 , 7 , 14 , 30 , 60 , 90 , 120 , 150 , 180 , 365 , 400 , 545 , 731 , 1827 , 2192 , 2557 , 2922 , 3288 , or 3653 . Additional resources What is Amazon CloudWatch Logs? (Amazon CloudWatch Logs User Guide) Specifying Credentials (AWS SDK for Go Developer Guide) Amazon CloudWatch Logs endpoints and quotas (AWS General Reference) 3.4.9. AWS EMF Exporter The AWS EMF Exporter converts the following OpenTelemetry metrics datapoints to the AWS CloudWatch Embedded Metric Format (EMF): Int64DataPoints DoubleDataPoints SummaryDataPoints The EMF metrics are then sent directly to the Amazon CloudWatch Logs service by using the PutLogEvents API. One of the benefits of using this exporter is the possibility to view logs and metrics in the Amazon CloudWatch console at https://console.aws.amazon.com/cloudwatch/ . Important The AWS EMF Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled AWS EMF Exporter # ... config: exporters: awsemf: log_group_name: "<group_name_of_amazon_cloudwatch_logs>" 1 log_stream_name: "<log_stream_of_amazon_cloudwatch_logs>" 2 resource_to_telemetry_conversion: 3 enabled: true region: <region> 4 endpoint: <endpoint> 5 log_retention: <supported_value_in_days> 6 namespace: <custom_namespace> 7 # ... 1 Customized log group name. 2 Customized log stream name. 3 Optional. Converts resource attributes to telemetry attributes such as metric labels. Disabled by default. 4 The AWS region of the log stream. If a region is not already set in the default credential provider chain, you must specify the region. 5 Optional. You can override the default Amazon CloudWatch Logs service endpoint to which the requests are forwarded. For the list of service endpoints by region, see Amazon CloudWatch Logs endpoints and quotas (AWS General Reference). 6 Optional. With this parameter, you can set the log retention policy for new Amazon CloudWatch log groups. If this parameter is omitted or set to 0 , the logs never expire by default. Supported values for retention in days are 1 , 3 , 5 , 7 , 14 , 30 , 60 , 90 , 120 , 150 , 180 , 365 , 400 , 545 , 731 , 1827 , 2192 , 2557 , 2922 , 3288 , or 3653 . 7 Optional. A custom namespace for the Amazon CloudWatch metrics. Log group name The log_group_name parameter allows you to customize the log group name and supports the default /metrics/default value or the following placeholders: /aws/metrics/{ClusterName} This placeholder is used to search for the ClusterName or aws.ecs.cluster.name resource attribute in the metrics data and replace it with the actual cluster name. {NodeName} This placeholder is used to search for the NodeName or k8s.node.name resource attribute. {TaskId} This placeholder is used to search for the TaskId or aws.ecs.task.id resource attribute. If no resource attribute is found in the resource attribute map, the placeholder is replaced by the undefined value. Log stream name The log_stream_name parameter allows you to customize the log stream name and supports the default otel-stream value or the following placeholders: {ClusterName} This placeholder is used to search for the ClusterName or aws.ecs.cluster.name resource attribute. {ContainerInstanceId} This placeholder is used to search for the ContainerInstanceId or aws.ecs.container.instance.id resource attribute. This resource attribute is valid only for the AWS ECS EC2 launch type. {NodeName} This placeholder is used to search for the NodeName or k8s.node.name resource attribute. {TaskDefinitionFamily} This placeholder is used to search for the TaskDefinitionFamily or aws.ecs.task.family resource attribute. {TaskId} This placeholder is used to search for the TaskId or aws.ecs.task.id resource attribute in the metrics data and replace it with the actual task ID. If no resource attribute is found in the resource attribute map, the placeholder is replaced by the undefined value. Additional resources Specification: Embedded metric format (Amazon CloudWatch User Guide) PutLogEvents (Amazon CloudWatch Logs API Reference) Amazon CloudWatch Logs endpoints and quotas (AWS General Reference) 3.4.10. AWS X-Ray Exporter The AWS X-Ray Exporter converts OpenTelemetry spans to AWS X-Ray Segment Documents and then sends them directly to the AWS X-Ray service. The AWS X-Ray Exporter uses the PutTraceSegments API and signs requests by using the AWS SDK for Go and the default credential provider chain. Important The AWS X-Ray Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled AWS X-Ray Exporter # ... config: exporters: awsxray: region: "<region>" 1 endpoint: <endpoint> 2 resource_arn: "<aws_resource_arn>" 3 role_arn: "<iam_role>" 4 indexed_attributes: [ "<indexed_attr_0>", "<indexed_attr_1>" ] 5 aws_log_groups: ["<group1>", "<group2>"] 6 request_timeout_seconds: 120 7 # ... 1 The destination region for the X-Ray segments sent to the AWS X-Ray service. For example, eu-west-1 . 2 Optional. You can override the default Amazon CloudWatch Logs service endpoint to which the requests are forwarded. For the list of service endpoints by region, see Amazon CloudWatch Logs endpoints and quotas (AWS General Reference). 3 The Amazon Resource Name (ARN) of the AWS resource that is running the Collector. 4 The AWS Identity and Access Management (IAM) role for uploading the X-Ray segments to a different account. 5 The list of attribute names to be converted to X-Ray annotations. 6 The list of log group names for Amazon CloudWatch Logs. 7 Time duration in seconds before timing out a request. If omitted, the default value is 30 . Additional resources What is AWS X-Ray? (AWS X-Ray Developer Guide) AWS SDK for Go API Reference (AWS Documentation) Specifying Credentials (AWS SDK for Go Developer Guide) IAM roles (AWS Identity and Access Management User Guide) 3.4.11. File Exporter The File Exporter writes telemetry data to files in persistent storage and supports file operations such as rotation, compression, and writing to multiple files. With this exporter, you can also use a resource attribute to control file naming. The only required setting is path , which specifies the destination path for telemetry files in the persistent-volume file system. Important The File Exporter is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the enabled File Exporter # ... config: | exporters: file: path: /data/metrics.json 1 rotation: 2 max_megabytes: 10 3 max_days: 3 4 max_backups: 3 5 localtime: true 6 format: proto 7 compression: zstd 8 flush_interval: 5 9 # ... 1 The file-system path where the data is to be written. There is no default. 2 File rotation is an optional feature of this exporter. By default, telemetry data is exported to a single file. Add the rotation setting to enable file rotation. 3 The max_megabytes setting is the maximum size a file is allowed to reach until it is rotated. The default is 100 . 4 The max_days setting is for how many days a file is to be retained, counting from the timestamp in the file name. There is no default. 5 The max_backups setting is for retaining several older files. The defalt is 100 . 6 The localtime setting specifies the local-time format for the timestamp, which is appended to the file name in front of any extension, when the file is rotated. The default is the Coordinated Universal Time (UTC). 7 The format for encoding the telemetry data before writing it to a file. The default format is json . The proto format is also supported. 8 File compression is optional and not set by default. This setting defines the compression algorithm for the data that is exported to a file. Currently, only the zstd compression algorithm is supported. There is no default. 9 The time interval between flushes. A value without a unit is set in nanoseconds. This setting is ignored when file rotation is enabled through the rotation settings. 3.4.12. Additional resources OpenTelemetry Protocol (OTLP) documentation 3.5. Connectors A connector connects two pipelines. It consumes data as an exporter at the end of one pipeline and emits data as a receiver at the start of another pipeline. It can consume and emit data of the same or different data type. It can generate and emit data to summarize the consumed data, or it can merely replicate or route data. Currently, the following General Availability and Technology Preview connectors are available for the Red Hat build of OpenTelemetry: Count Connector Routing Connector Forward Connector Spanmetrics Connector 3.5.1. Count Connector The Count Connector counts trace spans, trace span events, metrics, metric data points, and log records in exporter pipelines. Important The Count Connector is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . The following are the default metric names: trace.span.count trace.span.event.count metric.count metric.datapoint.count log.record.count You can also expose custom metric names. OpenTelemetry Collector custom resource (CR) with an enabled Count Connector # ... config: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 exporters: prometheus: endpoint: 0.0.0.0:8889 connectors: count: {} service: pipelines: 1 traces/in: receivers: [otlp] exporters: [count] 2 metrics/out: receivers: [count] 3 exporters: [prometheus] # ... 1 It is important to correctly configure the Count Connector as an exporter or receiver in the pipeline and to export the generated metrics to the correct exporter. 2 The Count Connector is configured to receive spans as an exporter. 3 The Count Connector is configured to emit generated metrics as a receiver. Tip If the Count Connector is not generating the expected metrics, you can check whether the OpenTelemetry Collector is receiving the expected spans, metrics, and logs, and whether the telemetry data flow through the Count Connector as expected. You can also use the Debug Exporter to inspect the incoming telemetry data. The Count Connector can count telemetry data according to defined conditions and expose those data as metrics when configured by using such fields as spans , spanevents , metrics , datapoints , or logs . See the example. Example OpenTelemetry Collector CR for the Count Connector to count spans by conditions # ... config: connectors: count: spans: 1 <custom_metric_name>: 2 description: "<custom_metric_description>" conditions: - 'attributes["env"] == "dev"' - 'name == "devevent"' # ... 1 In this example, the exposed metric counts spans with the specified conditions. 2 You can specify a custom metric name such as cluster.prod.event.count . Tip Write conditions correctly and follow the required syntax for attribute matching or telemetry field conditions. Improperly defined conditions are the most likely sources of errors. The Count Connector can count telemetry data according to defined attributes when configured by using such fields as spans , spanevents , metrics , datapoints , or logs . See the example. The attribute keys are injected into the telemetry data. You must define a value for the default_value field for missing attributes. Example OpenTelemetry Collector CR for the Count Connector to count logs by attributes # ... config: connectors: count: logs: 1 <custom_metric_name>: 2 description: "<custom_metric_description>" attributes: - key: env default_value: unknown 3 # ... 1 Specifies attributes for logs. 2 You can specify a custom metric name such as my.log.count . 3 Defines a default value when the attribute is not set. 3.5.2. Routing Connector The Routing Connector routes logs, metrics, and traces to specified pipelines according to resource attributes and their routing conditions, which are written as OpenTelemetry Transformation Language (OTTL) statements. Important The Routing Connector is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with an enabled Routing Connector # ... config: connectors: routing: table: 1 - statement: route() where attributes["X-Tenant"] == "dev" 2 pipelines: [traces/dev] 3 - statement: route() where attributes["X-Tenant"] == "prod" pipelines: [traces/prod] default_pipelines: [traces/dev] 4 error_mode: ignore 5 match_once: false 6 service: pipelines: traces/in: receivers: [otlp] exporters: [routing] traces/dev: receivers: [routing] exporters: [otlp/dev] traces/prod: receivers: [routing] exporters: [otlp/prod] # ... 1 Connector routing table. 2 Routing conditions written as OTTL statements. 3 Destination pipelines for routing the matching telemetry data. 4 Destination pipelines for routing the telemetry data for which no routing condition is satisfied. 5 Error-handling mode: The propagate value is for logging an error and dropping the payload. The ignore value is for ignoring the condition and attempting to match with the one. The silent value is the same as ignore but without logging the error. The default is propagate . 6 When set to true , the payload is routed only to the first pipeline whose routing condition is met. The default is false . 3.5.3. Forward Connector The Forward Connector merges two pipelines of the same type. Important The Forward Connector is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with an enabled Forward Connector # ... config: receivers: otlp: protocols: grpc: jaeger: protocols: grpc: processors: batch: exporters: otlp: endpoint: tempo-simplest-distributor:4317 tls: insecure: true connectors: forward: {} service: pipelines: traces/regiona: receivers: [otlp] processors: [] exporters: [forward] traces/regionb: receivers: [jaeger] processors: [] exporters: [forward] traces: receivers: [forward] processors: [batch] exporters: [otlp] # ... 3.5.4. Spanmetrics Connector The Spanmetrics Connector aggregates Request, Error, and Duration (R.E.D) OpenTelemetry metrics from span data. OpenTelemetry Collector custom resource with an enabled Spanmetrics Connector # ... config: connectors: spanmetrics: metrics_flush_interval: 15s 1 service: pipelines: traces: exporters: [spanmetrics] metrics: receivers: [spanmetrics] # ... 1 Defines the flush interval of the generated metrics. Defaults to 15s . 3.5.5. Additional resources OpenTelemetry Protocol (OTLP) documentation 3.6. Extensions Extensions add capabilities to the Collector. For example, authentication can be added to the receivers and exporters automatically. Currently, the following General Availability and Technology Preview extensions are available for the Red Hat build of OpenTelemetry: BearerTokenAuth Extension OAuth2Client Extension File Storage Extension OIDC Auth Extension Jaeger Remote Sampling Extension Performance Profiler Extension Health Check Extension zPages Extension 3.6.1. BearerTokenAuth Extension The BearerTokenAuth Extension is an authenticator for receivers and exporters that are based on the HTTP and the gRPC protocol. You can use the OpenTelemetry Collector custom resource to configure client authentication and server authentication for the BearerTokenAuth Extension on the receiver and exporter side. This extension supports traces, metrics, and logs. OpenTelemetry Collector custom resource with client and server authentication configured for the BearerTokenAuth Extension # ... config: extensions: bearertokenauth: scheme: "Bearer" 1 token: "<token>" 2 filename: "<token_file>" 3 receivers: otlp: protocols: http: auth: authenticator: bearertokenauth 4 exporters: otlp: auth: authenticator: bearertokenauth 5 service: extensions: [bearertokenauth] pipelines: traces: receivers: [otlp] exporters: [otlp] # ... 1 You can configure the BearerTokenAuth Extension to send a custom scheme . The default is Bearer . 2 You can add the BearerTokenAuth Extension token as metadata to identify a message. 3 Path to a file that contains an authorization token that is transmitted with every message. 4 You can assign the authenticator configuration to an OTLP Receiver. 5 You can assign the authenticator configuration to an OTLP Exporter. 3.6.2. OAuth2Client Extension The OAuth2Client Extension is an authenticator for exporters that are based on the HTTP and the gRPC protocol. Client authentication for the OAuth2Client Extension is configured in a separate section in the OpenTelemetry Collector custom resource. This extension supports traces, metrics, and logs. Important The OAuth2Client Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with client authentication configured for the OAuth2Client Extension # ... config: extensions: oauth2client: client_id: <client_id> 1 client_secret: <client_secret> 2 endpoint_params: 3 audience: <audience> token_url: https://example.com/oauth2/default/v1/token 4 scopes: ["api.metrics"] 5 # tls settings for the token client tls: 6 insecure: true 7 ca_file: /var/lib/mycert.pem 8 cert_file: <cert_file> 9 key_file: <key_file> 10 timeout: 2s 11 receivers: otlp: protocols: http: {} exporters: otlp: auth: authenticator: oauth2client 12 service: extensions: [oauth2client] pipelines: traces: receivers: [otlp] exporters: [otlp] # ... 1 Client identifier, which is provided by the identity provider. 2 Confidential key used to authenticate the client to the identity provider. 3 Further metadata, in the key-value pair format, which is transferred during authentication. For example, audience specifies the intended audience for the access token, indicating the recipient of the token. 4 The URL of the OAuth2 token endpoint, where the Collector requests access tokens. 5 The scopes define the specific permissions or access levels requested by the client. 6 The Transport Layer Security (TLS) settings for the token client, which is used to establish a secure connection when requesting tokens. 7 When set to true , configures the Collector to use an insecure or non-verified TLS connection to call the configured token endpoint. 8 The path to a Certificate Authority (CA) file that is used to verify the server's certificate during the TLS handshake. 9 The path to the client certificate file that the client must use to authenticate itself to the OAuth2 server if required. 10 The path to the client's private key file that is used with the client certificate if needed for authentication. 11 Sets a timeout for the token client's request. 12 You can assign the authenticator configuration to an OTLP exporter. 3.6.3. File Storage Extension The File Storage Extension supports traces, metrics, and logs. This extension can persist the state to the local file system. This extension persists the sending queue for the OpenTelemetry Protocol (OTLP) exporters that are based on the HTTP and the gRPC protocols. This extension requires the read and write access to a directory. This extension can use a default directory, but the default directory must already exist. Important The File Storage Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with a configured File Storage Extension that persists an OTLP sending queue # ... config: extensions: file_storage/all_settings: directory: /var/lib/otelcol/mydir 1 timeout: 1s 2 compaction: on_start: true 3 directory: /tmp/ 4 max_transaction_size: 65_536 5 fsync: false 6 exporters: otlp: sending_queue: storage: file_storage/all_settings 7 service: extensions: [file_storage/all_settings] 8 pipelines: traces: receivers: [otlp] exporters: [otlp] # ... 1 Specifies the directory in which the telemetry data is stored. 2 Specifies the timeout time interval for opening the stored files. 3 Starts compaction when the Collector starts. If omitted, the default is false . 4 Specifies the directory in which the compactor stores the telemetry data. 5 Defines the maximum size of the compaction transaction. To ignore the transaction size, set to zero. If omitted, the default is 65536 bytes. 6 When set, forces the database to perform an fsync call after each write operation. This helps to ensure database integrity if there is an interruption to the database process, but at the cost of performance. 7 Buffers the OTLP Exporter data on the local file system. 8 Starts the File Storage Extension by the Collector. 3.6.4. OIDC Auth Extension The OIDC Auth Extension authenticates incoming requests to receivers by using the OpenID Connect (OIDC) protocol. It validates the ID token in the authorization header against the issuer and updates the authentication context of the incoming request. Important The OIDC Auth Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the configured OIDC Auth Extension # ... config: extensions: oidc: attribute: authorization 1 issuer_url: https://example.com/auth/realms/opentelemetry 2 issuer_ca_path: /var/run/tls/issuer.pem 3 audience: otel-collector 4 username_claim: email 5 receivers: otlp: protocols: grpc: auth: authenticator: oidc exporters: debug: {} service: extensions: [oidc] pipelines: traces: receivers: [otlp] exporters: [debug] # ... 1 The name of the header that contains the ID token. The default name is authorization . 2 The base URL of the OIDC provider. 3 Optional: The path to the issuer's CA certificate. 4 The audience for the token. 5 The name of the claim that contains the username. The default name is sub . 3.6.5. Jaeger Remote Sampling Extension The Jaeger Remote Sampling Extension enables serving sampling strategies after Jaeger's remote sampling API. You can configure this extension to proxy requests to a backing remote sampling server such as a Jaeger collector down the pipeline or to a static JSON file from the local file system. Important The Jaeger Remote Sampling Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with a configured Jaeger Remote Sampling Extension # ... config: extensions: jaegerremotesampling: source: reload_interval: 30s 1 remote: endpoint: jaeger-collector:14250 2 file: /etc/otelcol/sampling_strategies.json 3 receivers: otlp: protocols: http: {} exporters: debug: {} service: extensions: [jaegerremotesampling] pipelines: traces: receivers: [otlp] exporters: [debug] # ... 1 The time interval at which the sampling configuration is updated. 2 The endpoint for reaching the Jaeger remote sampling strategy provider. 3 The path to a local file that contains a sampling strategy configuration in the JSON format. Example of a Jaeger Remote Sampling strategy file { "service_strategies": [ { "service": "foo", "type": "probabilistic", "param": 0.8, "operation_strategies": [ { "operation": "op1", "type": "probabilistic", "param": 0.2 }, { "operation": "op2", "type": "probabilistic", "param": 0.4 } ] }, { "service": "bar", "type": "ratelimiting", "param": 5 } ], "default_strategy": { "type": "probabilistic", "param": 0.5, "operation_strategies": [ { "operation": "/health", "type": "probabilistic", "param": 0.0 }, { "operation": "/metrics", "type": "probabilistic", "param": 0.0 } ] } } 3.6.6. Performance Profiler Extension The Performance Profiler Extension enables the Go net/http/pprof endpoint. Developers use this extension to collect performance profiles and investigate issues with the service. Important The Performance Profiler Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the configured Performance Profiler Extension # ... config: extensions: pprof: endpoint: localhost:1777 1 block_profile_fraction: 0 2 mutex_profile_fraction: 0 3 save_to_file: test.pprof 4 receivers: otlp: protocols: http: {} exporters: debug: {} service: extensions: [pprof] pipelines: traces: receivers: [otlp] exporters: [debug] # ... 1 The endpoint at which this extension listens. Use localhost: to make it available only locally or ":" to make it available on all network interfaces. The default value is localhost:1777 . 2 Sets a fraction of blocking events to be profiled. To disable profiling, set this to 0 or a negative integer. See the documentation for the runtime package. The default value is 0 . 3 Set a fraction of mutex contention events to be profiled. To disable profiling, set this to 0 or a negative integer. See the documentation for the runtime package. The default value is 0 . 4 The name of the file in which the CPU profile is to be saved. Profiling starts when the Collector starts. Profiling is saved to the file when the Collector is terminated. 3.6.7. Health Check Extension The Health Check Extension provides an HTTP URL for checking the status of the OpenTelemetry Collector. You can use this extension as a liveness and readiness probe on OpenShift. Important The Health Check Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the configured Health Check Extension # ... config: extensions: health_check: endpoint: "0.0.0.0:13133" 1 tls: 2 ca_file: "/path/to/ca.crt" cert_file: "/path/to/cert.crt" key_file: "/path/to/key.key" path: "/health/status" 3 check_collector_pipeline: 4 enabled: true 5 interval: "5m" 6 exporter_failure_threshold: 5 7 receivers: otlp: protocols: http: {} exporters: debug: {} service: extensions: [health_check] pipelines: traces: receivers: [otlp] exporters: [debug] # ... 1 The target IP address for publishing the health check status. The default is 0.0.0.0:13133 . 2 The TLS server-side configuration. Defines paths to TLS certificates. If omitted, the TLS is disabled. 3 The path for the health check server. The default is / . 4 Settings for the Collector pipeline health check. 5 Enables the Collector pipeline health check. The default is false . 6 The time interval for checking the number of failures. The default is 5m . 7 The threshold of multiple failures until which a container is still marked as healthy. The default is 5 . 3.6.8. zPages Extension The zPages Extension provides an HTTP endpoint that serves live data for debugging instrumented components in real time. You can use this extension for in-process diagnostics and insights into traces and metrics without relying on an external backend. With this extension, you can monitor and troubleshoot the behavior of the OpenTelemetry Collector and related components by watching the diagnostic information at the provided endpoint. Important The zPages Extension is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . OpenTelemetry Collector custom resource with the configured zPages Extension # ... config: extensions: zpages: endpoint: "localhost:55679" 1 receivers: otlp: protocols: http: {} exporters: debug: {} service: extensions: [zpages] pipelines: traces: receivers: [otlp] exporters: [debug] # ... 1 Specifies the HTTP endpoint for serving the zPages extension. The default is localhost:55679 . Important Accessing the HTTP endpoint requires port-forwarding because the Red Hat build of OpenTelemetry Operator does not expose this route. You can enable port-forwarding by running the following oc command: USD oc port-forward pod/USD(oc get pod -l app.kubernetes.io/name=instance-collector -o=jsonpath='{.items[0].metadata.name}') 55679 The Collector provides the following zPages for diagnostics: ServiceZ Shows an overview of the Collector services and links to the following zPages: PipelineZ , ExtensionZ , and FeatureZ . This page also displays information about the build version and runtime. An example of this page's URL is http://localhost:55679/debug/servicez . PipelineZ Shows detailed information about the active pipelines in the Collector. This page displays the pipeline type, whether data are modified, and the associated receivers, processors, and exporters for each pipeline. An example of this page's URL is http://localhost:55679/debug/pipelinez . ExtensionZ Shows the currently active extensions in the Collector. An example of this page's URL is http://localhost:55679/debug/extensionz . FeatureZ Shows the feature gates enabled in the Collector along with their status and description. An example of this page's URL is http://localhost:55679/debug/featurez . TraceZ Shows spans categorized by latency. Available time ranges include 0 ms, 10 ms, 100 ms, 1 ms, 10 ms, 100 ms, 1 s, 10 s, 1 m. This page also allows for quick inspection of error samples. An example of this page's URL is http://localhost:55679/debug/tracez . 3.6.9. Additional resources OpenTelemetry Protocol (OTLP) documentation 3.7. Target Allocator The Target Allocator is an optional component of the OpenTelemetry Operator that shards scrape targets across the deployed fleet of OpenTelemetry Collector instances. The Target Allocator integrates with the Prometheus PodMonitor and ServiceMonitor custom resources (CR). When the Target Allocator is enabled, the OpenTelemetry Operator adds the http_sd_config field to the enabled prometheus receiver that connects to the Target Allocator service. Important The Target Allocator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Example OpenTelemetryCollector CR with the enabled Target Allocator apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otel namespace: observability spec: mode: statefulset 1 targetAllocator: enabled: true 2 serviceAccount: 3 prometheusCR: enabled: true 4 scrapeInterval: 10s serviceMonitorSelector: 5 name: app1 podMonitorSelector: 6 name: app2 config: receivers: prometheus: 7 config: scrape_configs: [] processors: exporters: debug: {} service: pipelines: metrics: receivers: [prometheus] processors: [] exporters: [debug] # ... 1 When the Target Allocator is enabled, the deployment mode must be set to statefulset . 2 Enables the Target Allocator. Defaults to false . 3 The service account name of the Target Allocator deployment. The service account needs to have RBAC to get the ServiceMonitor , PodMonitor custom resources, and other objects from the cluster to properly set labels on scraped metrics. The default service name is <collector_name>-targetallocator . 4 Enables integration with the Prometheus PodMonitor and ServiceMonitor custom resources. 5 Label selector for the Prometheus ServiceMonitor custom resources. When left empty, enables all service monitors. 6 Label selector for the Prometheus PodMonitor custom resources. When left empty, enables all pod monitors. 7 Prometheus receiver with the minimal, empty scrape_config: [] configuration option. The Target Allocator deployment uses the Kubernetes API to get relevant objects from the cluster, so it requires a custom RBAC configuration. RBAC configuration for the Target Allocator service account apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-targetallocator rules: - apiGroups: [""] resources: - services - pods - namespaces verbs: ["get", "list", "watch"] - apiGroups: ["monitoring.coreos.com"] resources: - servicemonitors - podmonitors - scrapeconfigs - probes verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: - endpointslices verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-targetallocator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: otel-targetallocator subjects: - kind: ServiceAccount name: otel-targetallocator 1 namespace: observability 2 # ... 1 The name of the Target Allocator service account mane. 2 The namespace of the Target Allocator service account.
apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: cluster-collector namespace: tracing-system spec: mode: deployment observability: metrics: enableMetrics: true config: receivers: otlp: protocols: grpc: {} http: {} processors: {} exporters: otlp: endpoint: otel-collector-headless.tracing-system.svc:4317 tls: ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt" prometheus: endpoint: 0.0.0.0:8889 resource_to_telemetry_conversion: enabled: true # by default resource attributes are dropped service: 1 pipelines: traces: receivers: [otlp] processors: [] exporters: [otlp] metrics: receivers: [otlp] processors: [] exporters: [prometheus] receivers: processors: exporters: connectors: extensions: service: pipelines: service: pipelines: traces: receivers: service: pipelines: traces: processors: service: pipelines: traces: exporters: service: pipelines: metrics: receivers: service: pipelines: metrics: processors: service: pipelines: metrics: exporters: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: generate-processors-rbac rules: - apiGroups: - rbac.authorization.k8s.io resources: - clusterrolebindings - clusterroles verbs: - create - delete - get - list - patch - update - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: generate-processors-rbac roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: generate-processors-rbac subjects: - kind: ServiceAccount name: opentelemetry-operator-controller-manager namespace: openshift-opentelemetry-operator config: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem client_ca_file: client.pem 3 reload_interval: 1h 4 http: endpoint: 0.0.0.0:4318 5 tls: {} 6 service: pipelines: traces: receivers: [otlp] metrics: receivers: [otlp] config: receivers: jaeger: protocols: grpc: endpoint: 0.0.0.0:14250 1 thrift_http: endpoint: 0.0.0.0:14268 2 thrift_compact: endpoint: 0.0.0.0:6831 3 thrift_binary: endpoint: 0.0.0.0:6832 4 tls: {} 5 service: pipelines: traces: receivers: [jaeger] apiVersion: v1 kind: ServiceAccount metadata: name: otel-hostfs-daemonset namespace: <namespace> --- apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints allowHostDirVolumePlugin: true allowHostIPC: false allowHostNetwork: false allowHostPID: true allowHostPorts: false allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: null defaultAddCapabilities: - SYS_ADMIN fsGroup: type: RunAsAny groups: [] metadata: name: otel-hostmetrics readOnlyRootFilesystem: true runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny supplementalGroups: type: RunAsAny users: - system:serviceaccount:<namespace>:otel-hostfs-daemonset volumes: - configMap - emptyDir - hostPath - projected --- apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otel namespace: <namespace> spec: serviceAccount: otel-hostfs-daemonset mode: daemonset volumeMounts: - mountPath: /hostfs name: host readOnly: true volumes: - hostPath: path: / name: host config: receivers: hostmetrics: collection_interval: 10s 1 initial_delay: 1s 2 root_path: / 3 scrapers: 4 cpu: {} memory: {} disk: {} service: pipelines: metrics: receivers: [hostmetrics] apiVersion: v1 kind: ServiceAccount metadata: name: otel-k8sobj namespace: <namespace> --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-k8sobj namespace: <namespace> rules: - apiGroups: - "" resources: - events - pods verbs: - get - list - watch - apiGroups: - "events.k8s.io" resources: - events verbs: - watch - list --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-k8sobj subjects: - kind: ServiceAccount name: otel-k8sobj namespace: <namespace> roleRef: kind: ClusterRole name: otel-k8sobj apiGroup: rbac.authorization.k8s.io --- apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otel-k8s-obj namespace: <namespace> spec: serviceAccount: otel-k8sobj mode: deployment config: receivers: k8sobjects: auth_type: serviceAccount objects: - name: pods 1 mode: pull 2 interval: 30s 3 label_selector: 4 field_selector: 5 namespaces: [<namespace>,...] 6 - name: events mode: watch exporters: debug: service: pipelines: logs: receivers: [k8sobjects] exporters: [debug] config: receivers: kubeletstats: collection_interval: 20s auth_type: "serviceAccount" endpoint: "https://USD{env:K8S_NODE_NAME}:10250" insecure_skip_verify: true service: pipelines: metrics: receivers: [kubeletstats] env: - name: K8S_NODE_NAME 1 valueFrom: fieldRef: fieldPath: spec.nodeName apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: [''] resources: ['nodes/stats'] verbs: ['get', 'watch', 'list'] - apiGroups: [""] resources: ["nodes/proxy"] 1 verbs: ["get"] config: receivers: prometheus: config: scrape_configs: 1 - job_name: 'my-app' 2 scrape_interval: 5s 3 static_configs: - targets: ['my-app.example.svc.cluster.local:8888'] 4 service: pipelines: metrics: receivers: [prometheus] config: otlpjsonfile: include: - "/var/log/*.log" 1 exclude: - "/var/log/test.log" 2 config: receivers: zipkin: endpoint: 0.0.0.0:9411 1 tls: {} 2 service: pipelines: traces: receivers: [zipkin] config: receivers: kafka: brokers: ["localhost:9092"] 1 protocol_version: 2.0.0 2 topic: otlp_spans 3 auth: plain_text: 4 username: example password: example tls: 5 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 6 server_name_override: kafka.example.corp 7 service: pipelines: traces: receivers: [kafka] config: receivers: k8s_cluster: distribution: openshift collection_interval: 10s exporters: debug: {} service: pipelines: metrics: receivers: [k8s_cluster] exporters: [debug] logs/entity_events: receivers: [k8s_cluster] exporters: [debug] apiVersion: v1 kind: ServiceAccount metadata: labels: app: otelcontribcol name: otelcontribcol apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otelcontribcol labels: app: otelcontribcol rules: - apiGroups: - quota.openshift.io resources: - clusterresourcequotas verbs: - get - list - watch - apiGroups: - "" resources: - events - namespaces - namespaces/status - nodes - nodes/spec - pods - pods/status - replicationcontrollers - replicationcontrollers/status - resourcequotas - services verbs: - get - list - watch - apiGroups: - apps resources: - daemonsets - deployments - replicasets - statefulsets verbs: - get - list - watch - apiGroups: - extensions resources: - daemonsets - deployments - replicasets verbs: - get - list - watch - apiGroups: - batch resources: - jobs - cronjobs verbs: - get - list - watch - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - get - list - watch apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otelcontribcol labels: app: otelcontribcol roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: otelcontribcol subjects: - kind: ServiceAccount name: otelcontribcol namespace: default config: receivers: opencensus: endpoint: 0.0.0.0:9411 1 tls: 2 cors_allowed_origins: 3 - https://*.<example>.com service: pipelines: traces: receivers: [opencensus] config: receivers: filelog: include: [ /simple.log ] 1 operators: 2 - type: regex_parser regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)USD' timestamp: parse_from: attributes.time layout: '%Y-%m-%d %H:%M:%S' severity: parse_from: attributes.sev apiVersion: v1 kind: Namespace metadata: name: otel-journald labels: security.openshift.io/scc.podSecurityLabelSync: "false" pod-security.kubernetes.io/enforce: "privileged" pod-security.kubernetes.io/audit: "privileged" pod-security.kubernetes.io/warn: "privileged" --- apiVersion: v1 kind: ServiceAccount metadata: name: privileged-sa namespace: otel-journald --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-journald-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:openshift:scc:privileged subjects: - kind: ServiceAccount name: privileged-sa namespace: otel-journald --- apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otel-journald-logs namespace: otel-journald spec: mode: daemonset serviceAccount: privileged-sa securityContext: allowPrivilegeEscalation: false capabilities: drop: - CHOWN - DAC_OVERRIDE - FOWNER - FSETID - KILL - NET_BIND_SERVICE - SETGID - SETPCAP - SETUID readOnlyRootFilesystem: true seLinuxOptions: type: spc_t seccompProfile: type: RuntimeDefault config: receivers: journald: files: /var/log/journal/*/* priority: info 1 units: 2 - kubelet - crio - init.scope - dnsmasq all: true 3 retry_on_failure: enabled: true 4 initial_interval: 1s 5 max_interval: 30s 6 max_elapsed_time: 5m 7 processors: exporters: debug: {} service: pipelines: logs: receivers: [journald] exporters: [debug] volumeMounts: - name: journal-logs mountPath: /var/log/journal/ readOnly: true volumes: - name: journal-logs hostPath: path: /var/log/journal tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-collector labels: app: otel-collector rules: - apiGroups: - "" resources: - events - namespaces - namespaces/status - nodes - nodes/spec - pods - pods/status - replicationcontrollers - replicationcontrollers/status - resourcequotas - services verbs: - get - list - watch - apiGroups: - apps resources: - daemonsets - deployments - replicasets - statefulsets verbs: - get - list - watch - apiGroups: - extensions resources: - daemonsets - deployments - replicasets verbs: - get - list - watch - apiGroups: - batch resources: - jobs - cronjobs verbs: - get - list - watch - apiGroups: - autoscaling resources: - horizontalpodautoscalers verbs: - get - list - watch serviceAccount: otel-collector 1 config: receivers: k8s_events: namespaces: [project1, project2] 2 service: pipelines: logs: receivers: [k8s_events] config: processors: batch: timeout: 5s send_batch_max_size: 10000 service: pipelines: traces: processors: [batch] metrics: processors: [batch] config: processors: memory_limiter: check_interval: 1s limit_mib: 4000 spike_limit_mib: 800 service: pipelines: traces: processors: [batch] metrics: processors: [batch] kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: ["config.openshift.io"] resources: ["infrastructures", "infrastructures/status"] verbs: ["get", "watch", "list"] config: processors: resourcedetection: detectors: [openshift] override: true service: pipelines: traces: processors: [resourcedetection] metrics: processors: [resourcedetection] config: processors: resourcedetection/env: detectors: [env] 1 timeout: 2s override: false config: processors: attributes/example: actions: - key: db.table action: delete - key: redacted_span value: true action: upsert - key: copy_key from_attribute: key_original action: update - key: account_id value: 2245 action: insert - key: account_password action: delete - key: account_email action: hash - key: http.status_code action: convert converted_type: int config: processors: attributes: - key: cloud.availability_zone value: "zone-1" action: upsert - key: k8s.cluster.name from_attribute: k8s-cluster action: insert - key: redundant-attribute action: delete config: processors: span: name: from_attributes: [<key1>, <key2>, ...] 1 separator: <value> 2 config: processors: span/to_attributes: name: to_attributes: rules: - ^\/api\/v1\/document\/(?P<documentId>.*)\/updateUSD 1 config: processors: span/set_status: status: code: Error description: "<error_description>" kind: ClusterRole metadata: name: otel-collector rules: - apiGroups: [''] resources: ['pods', 'namespaces'] verbs: ['get', 'watch', 'list'] config: processors: k8sattributes: filter: node_from_env_var: KUBE_NODE_NAME config: processors: filter/ottl: error_mode: ignore 1 traces: span: - 'attributes["container.name"] == "app_container_1"' 2 - 'resource.attributes["host.name"] == "localhost"' 3 config: processors: routing: from_attribute: X-Tenant 1 default_exporters: 2 - jaeger table: 3 - value: acme exporters: [jaeger/acme] exporters: jaeger: endpoint: localhost:14250 jaeger/acme: endpoint: localhost:24250 config: processors: cumulativetodelta: include: 1 match_type: strict 2 metrics: 3 - <metric_1_name> - <metric_2_name> exclude: 4 match_type: regexp metrics: - "<regular_expression_for_metric_names>" config: processors: groupbyattrs: keys: 1 - <key1> 2 - <key2> config: processors: transform: error_mode: ignore 1 <trace|metric|log>_statements: 2 - context: <string> 3 conditions: 4 - <string> - <string> statements: 5 - <string> - <string> - <string> - context: <string> statements: - <string> - <string> - <string> config: transform: error_mode: ignore trace_statements: 1 - context: resource statements: - keep_keys(attributes, ["service.name", "service.namespace", "cloud.region", "process.command_line"]) 2 - replace_pattern(attributes["process.command_line"], "password\\=[^\\s]*(\\s?)", "password=***") 3 - limit(attributes, 100, []) - truncate_all(attributes, 4096) - context: span 4 statements: - set(status.code, 1) where attributes["http.path"] == "/health" - set(name, attributes["http.route"]) - replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}") - limit(attributes, 100, []) - truncate_all(attributes, 4096) config: exporters: otlp: endpoint: tempo-ingester:4317 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 3 insecure_skip_verify: false # 4 reload_interval: 1h 5 server_name_override: <name> 6 headers: 7 X-Scope-OrgID: "dev" service: pipelines: traces: exporters: [otlp] metrics: exporters: [otlp] config: exporters: otlphttp: endpoint: http://tempo-ingester:4318 1 tls: 2 headers: 3 X-Scope-OrgID: "dev" disable_keep_alives: false 4 service: pipelines: traces: exporters: [otlphttp] metrics: exporters: [otlphttp] config: exporters: debug: verbosity: detailed 1 sampling_initial: 5 2 sampling_thereafter: 200 3 use_internal_logger: true 4 service: pipelines: traces: exporters: [debug] metrics: exporters: [debug] config: exporters: loadbalancing: routing_key: "service" 1 protocol: otlp: 2 timeout: 1s resolver: 3 static: 4 hostnames: - backend-1:4317 - backend-2:4317 dns: 5 hostname: otelcol-headless.observability.svc.cluster.local k8s: 6 service: lb-svc.kube-public ports: - 15317 - 16317 config: exporters: prometheus: endpoint: 0.0.0.0:8889 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem namespace: prefix 3 const_labels: 4 label1: value1 enable_open_metrics: true 5 resource_to_telemetry_conversion: 6 enabled: true metric_expiration: 180m 7 add_metric_suffixes: false 8 service: pipelines: metrics: exporters: [prometheus] config: exporters: prometheusremotewrite: endpoint: "https://my-prometheus:7900/api/v1/push" 1 tls: 2 ca_file: ca.pem cert_file: cert.pem key_file: key.pem target_info: true 3 export_created_metric: true 4 max_batch_size_bytes: 3000000 5 service: pipelines: metrics: exporters: [prometheusremotewrite] config: exporters: kafka: brokers: ["localhost:9092"] 1 protocol_version: 2.0.0 2 topic: otlp_spans 3 auth: plain_text: 4 username: example password: example tls: 5 ca_file: ca.pem cert_file: cert.pem key_file: key.pem insecure: false 6 server_name_override: kafka.example.corp 7 service: pipelines: traces: exporters: [kafka] config: exporters: awscloudwatchlogs: log_group_name: "<group_name_of_amazon_cloudwatch_logs>" 1 log_stream_name: "<log_stream_of_amazon_cloudwatch_logs>" 2 region: <aws_region_of_log_stream> 3 endpoint: <service_endpoint_of_amazon_cloudwatch_logs> 4 log_retention: <supported_value_in_days> 5 config: exporters: awsemf: log_group_name: "<group_name_of_amazon_cloudwatch_logs>" 1 log_stream_name: "<log_stream_of_amazon_cloudwatch_logs>" 2 resource_to_telemetry_conversion: 3 enabled: true region: <region> 4 endpoint: <endpoint> 5 log_retention: <supported_value_in_days> 6 namespace: <custom_namespace> 7 config: exporters: awsxray: region: "<region>" 1 endpoint: <endpoint> 2 resource_arn: "<aws_resource_arn>" 3 role_arn: "<iam_role>" 4 indexed_attributes: [ "<indexed_attr_0>", "<indexed_attr_1>" ] 5 aws_log_groups: ["<group1>", "<group2>"] 6 request_timeout_seconds: 120 7 config: | exporters: file: path: /data/metrics.json 1 rotation: 2 max_megabytes: 10 3 max_days: 3 4 max_backups: 3 5 localtime: true 6 format: proto 7 compression: zstd 8 flush_interval: 5 9 config: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 exporters: prometheus: endpoint: 0.0.0.0:8889 connectors: count: {} service: pipelines: 1 traces/in: receivers: [otlp] exporters: [count] 2 metrics/out: receivers: [count] 3 exporters: [prometheus] config: connectors: count: spans: 1 <custom_metric_name>: 2 description: "<custom_metric_description>" conditions: - 'attributes["env"] == "dev"' - 'name == "devevent"' config: connectors: count: logs: 1 <custom_metric_name>: 2 description: "<custom_metric_description>" attributes: - key: env default_value: unknown 3 config: connectors: routing: table: 1 - statement: route() where attributes["X-Tenant"] == "dev" 2 pipelines: [traces/dev] 3 - statement: route() where attributes["X-Tenant"] == "prod" pipelines: [traces/prod] default_pipelines: [traces/dev] 4 error_mode: ignore 5 match_once: false 6 service: pipelines: traces/in: receivers: [otlp] exporters: [routing] traces/dev: receivers: [routing] exporters: [otlp/dev] traces/prod: receivers: [routing] exporters: [otlp/prod] config: receivers: otlp: protocols: grpc: jaeger: protocols: grpc: processors: batch: exporters: otlp: endpoint: tempo-simplest-distributor:4317 tls: insecure: true connectors: forward: {} service: pipelines: traces/regiona: receivers: [otlp] processors: [] exporters: [forward] traces/regionb: receivers: [jaeger] processors: [] exporters: [forward] traces: receivers: [forward] processors: [batch] exporters: [otlp] config: connectors: spanmetrics: metrics_flush_interval: 15s 1 service: pipelines: traces: exporters: [spanmetrics] metrics: receivers: [spanmetrics] config: extensions: bearertokenauth: scheme: "Bearer" 1 token: "<token>" 2 filename: "<token_file>" 3 receivers: otlp: protocols: http: auth: authenticator: bearertokenauth 4 exporters: otlp: auth: authenticator: bearertokenauth 5 service: extensions: [bearertokenauth] pipelines: traces: receivers: [otlp] exporters: [otlp] config: extensions: oauth2client: client_id: <client_id> 1 client_secret: <client_secret> 2 endpoint_params: 3 audience: <audience> token_url: https://example.com/oauth2/default/v1/token 4 scopes: ["api.metrics"] 5 # tls settings for the token client tls: 6 insecure: true 7 ca_file: /var/lib/mycert.pem 8 cert_file: <cert_file> 9 key_file: <key_file> 10 timeout: 2s 11 receivers: otlp: protocols: http: {} exporters: otlp: auth: authenticator: oauth2client 12 service: extensions: [oauth2client] pipelines: traces: receivers: [otlp] exporters: [otlp] config: extensions: file_storage/all_settings: directory: /var/lib/otelcol/mydir 1 timeout: 1s 2 compaction: on_start: true 3 directory: /tmp/ 4 max_transaction_size: 65_536 5 fsync: false 6 exporters: otlp: sending_queue: storage: file_storage/all_settings 7 service: extensions: [file_storage/all_settings] 8 pipelines: traces: receivers: [otlp] exporters: [otlp] config: extensions: oidc: attribute: authorization 1 issuer_url: https://example.com/auth/realms/opentelemetry 2 issuer_ca_path: /var/run/tls/issuer.pem 3 audience: otel-collector 4 username_claim: email 5 receivers: otlp: protocols: grpc: auth: authenticator: oidc exporters: debug: {} service: extensions: [oidc] pipelines: traces: receivers: [otlp] exporters: [debug] config: extensions: jaegerremotesampling: source: reload_interval: 30s 1 remote: endpoint: jaeger-collector:14250 2 file: /etc/otelcol/sampling_strategies.json 3 receivers: otlp: protocols: http: {} exporters: debug: {} service: extensions: [jaegerremotesampling] pipelines: traces: receivers: [otlp] exporters: [debug] { "service_strategies": [ { "service": "foo", "type": "probabilistic", "param": 0.8, "operation_strategies": [ { "operation": "op1", "type": "probabilistic", "param": 0.2 }, { "operation": "op2", "type": "probabilistic", "param": 0.4 } ] }, { "service": "bar", "type": "ratelimiting", "param": 5 } ], "default_strategy": { "type": "probabilistic", "param": 0.5, "operation_strategies": [ { "operation": "/health", "type": "probabilistic", "param": 0.0 }, { "operation": "/metrics", "type": "probabilistic", "param": 0.0 } ] } } config: extensions: pprof: endpoint: localhost:1777 1 block_profile_fraction: 0 2 mutex_profile_fraction: 0 3 save_to_file: test.pprof 4 receivers: otlp: protocols: http: {} exporters: debug: {} service: extensions: [pprof] pipelines: traces: receivers: [otlp] exporters: [debug] config: extensions: health_check: endpoint: "0.0.0.0:13133" 1 tls: 2 ca_file: "/path/to/ca.crt" cert_file: "/path/to/cert.crt" key_file: "/path/to/key.key" path: "/health/status" 3 check_collector_pipeline: 4 enabled: true 5 interval: "5m" 6 exporter_failure_threshold: 5 7 receivers: otlp: protocols: http: {} exporters: debug: {} service: extensions: [health_check] pipelines: traces: receivers: [otlp] exporters: [debug] config: extensions: zpages: endpoint: "localhost:55679" 1 receivers: otlp: protocols: http: {} exporters: debug: {} service: extensions: [zpages] pipelines: traces: receivers: [otlp] exporters: [debug] oc port-forward pod/USD(oc get pod -l app.kubernetes.io/name=instance-collector -o=jsonpath='{.items[0].metadata.name}') 55679 apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otel namespace: observability spec: mode: statefulset 1 targetAllocator: enabled: true 2 serviceAccount: 3 prometheusCR: enabled: true 4 scrapeInterval: 10s serviceMonitorSelector: 5 name: app1 podMonitorSelector: 6 name: app2 config: receivers: prometheus: 7 config: scrape_configs: [] processors: exporters: debug: {} service: pipelines: metrics: receivers: [prometheus] processors: [] exporters: [debug] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: otel-targetallocator rules: - apiGroups: [""] resources: - services - pods - namespaces verbs: ["get", "list", "watch"] - apiGroups: ["monitoring.coreos.com"] resources: - servicemonitors - podmonitors - scrapeconfigs - probes verbs: ["get", "list", "watch"] - apiGroups: ["discovery.k8s.io"] resources: - endpointslices verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otel-targetallocator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: otel-targetallocator subjects: - kind: ServiceAccount name: otel-targetallocator 1 namespace: observability 2