Chapter 20. Setting up metrics and dashboards for AMQ Streams
Chapter 20. Setting up metrics and dashboards for AMQ Streams Collecting metrics is critical for understanding the health and performance of your Kafka deployment. By monitoring metrics, you can actively identify issues before they become critical and make informed decisions about resource allocation and capacity planning. Without metrics, you may be left with limited visibility into the behavior of your Kafka deployment, which can make troubleshooting more difficult and time-consuming. Setting up metrics can save you time and resources in the long run, and help ensure the reliability of your Kafka deployment. Metrics are available for each component in AMQ Streams, providing valuable insights into their individual performance. While other components require configuration to expose metrics metric exposure, AMQ Streams operators automatically expose Prometheus metrics by default. These metrics include: Reconciliation count Custom Resource count being processed Reconciliation duration JVM metrics You can also collect metrics specific to oauth authentication and opa or keycloak authorization by enabling the enableMetrics property in the listener or authorization configuration of the Kafka resource. Similarly, you can enable metrics for oauth authentication in custom resources such as KafkaBridge , KafkaConnect , KafkaMirrorMaker , and KafkaMirrorMaker2 . You can use Prometheus and Grafana to monitor AMQ Streams. Prometheus consumes metrics from the running pods in your cluster when configured with Prometheus rules. Grafana visualizes these metrics on dashboards, providing an intuitive interface for monitoring. To facilitate metrics integration, AMQ Streams provides example Prometheus rules and Grafana dashboards for AMQ Streams components. You can customize the example Grafana dashboards to suit your specific deployment requirements. You can use rules to define conditions that trigger alerts based on specific metrics. Depending on your monitoring requirements, you can do the following: Set up and deploy Prometheus to expose metrics Deploy Kafka Exporter to provide additional metrics Use Grafana to present the Prometheus metrics Additionally, you can configure your deployment to track messages end-to-end by setting up distributed tracing , or retrieve troubleshooting data using the diagnostics tool ( report.sh ). Note AMQ Streams provides example installation files for Prometheus and Grafana, which can serve as a starting point for monitoring your AMQ Streams deployment. For further support, try engaging with the Prometheus and Grafana developer communities. Supporting documentation for metrics and monitoring tools For more information on the metrics and monitoring tools, refer to the supporting documentation: Prometheus Prometheus configuration Kafka Exporter Grafana Labs Apache Kafka Monitoring describes JMX metrics exposed by Apache Kafka ZooKeeper JMX describes JMX metrics exposed by Apache ZooKeeper 20.1. Monitoring consumer lag with Kafka Exporter Kafka Exporter is an open source project to enhance monitoring of Apache Kafka brokers and clients. You can configure the Kafka resource to deploy Kafka Exporter with your Kafka cluster . Kafka Exporter extracts additional metrics data from Kafka brokers related to offsets, consumer groups, consumer lag, and topics. The metrics data is used, for example, to help identify slow consumers. Lag data is exposed as Prometheus metrics, which can then be presented in Grafana for analysis. Kafka Exporter reads from the __consumer_offsets topic, which stores information on committed offsets for consumer groups. For Kafka Exporter to be able to work properly, consumer groups needs to be in use. A Grafana dashboard for Kafka Exporter is one of a number of example Grafana dashboards provided by AMQ Streams. Important Kafka Exporter provides only additional metrics related to consumer lag and consumer offsets. For regular Kafka metrics, you have to configure the Prometheus metrics in Kafka brokers . Consumer lag indicates the difference in the rate of production and consumption of messages. Specifically, consumer lag for a given consumer group indicates the delay between the last message in the partition and the message being currently picked up by that consumer. The lag reflects the position of the consumer offset in relation to the end of the partition log. Consumer lag between the producer and consumer offset This difference is sometimes referred to as the delta between the producer offset and consumer offset: the read and write positions in the Kafka broker topic partitions. Suppose a topic streams 100 messages a second. A lag of 1000 messages between the producer offset (the topic partition head) and the last offset the consumer has read means a 10-second delay. The importance of monitoring consumer lag For applications that rely on the processing of (near) real-time data, it is critical to monitor consumer lag to check that it does not become too big. The greater the lag becomes, the further the process moves from the real-time processing objective. Consumer lag, for example, might be a result of consuming too much old data that has not been purged, or through unplanned shutdowns. Reducing consumer lag Use the Grafana charts to analyze lag and to check if actions to reduce lag are having an impact on an affected consumer group. If, for example, Kafka brokers are adjusted to reduce lag, the dashboard will show the Lag by consumer group chart going down and the Messages consumed per minute chart going up. Typical actions to reduce lag include: Scaling-up consumer groups by adding new consumers Increasing the retention time for a message to remain in a topic Adding more disk capacity to increase the message buffer Actions to reduce consumer lag depend on the underlying infrastructure and the use cases AMQ Streams is supporting. For instance, a lagging consumer is less likely to benefit from the broker being able to service a fetch request from its disk cache. And in certain cases, it might be acceptable to automatically drop messages until a consumer has caught up. 20.2. Monitoring Cruise Control operations Cruise Control monitors Kafka brokers in order to track the utilization of brokers, topics, and partitions. Cruise Control also provides a set of metrics for monitoring its own performance. The Cruise Control metrics reporter collects raw metrics data from Kafka brokers. The data is produced to topics that are automatically created by Cruise Control. The metrics are used to generate optimization proposals for Kafka clusters . Cruise Control metrics are available for real-time monitoring of Cruise Control operations. For example, you can use Cruise Control metrics to monitor the status of rebalancing operations that are running or provide alerts on any anomalies that are detected in an operation's performance. You expose Cruise Control metrics by enabling the Prometheus JMX Exporter in the Cruise Control configuration . Note For a full list of available Cruise Control metrics, which are known as sensors , see the Cruise Control documentation 20.2.1. Monitoring balancedness scores Cruise Control metrics include a balancedness score. Balancedness is the measure of how evenly a workload is distributed in a Kafka cluster. The Cruise Control metric for balancedness score ( balancedness-score ) might differ from the balancedness score in the KafkaRebalance resource. Cruise Control calculates each score using anomaly.detection.goals which might not be the same as the default.goals used in the KafkaRebalance resource. The anomaly.detection.goals are specified in the spec.cruiseControl.config of the Kafka custom resource. Note Refreshing the KafkaRebalance resource fetches an optimization proposal. The latest cached optimization proposal is fetched if one of the following conditions applies: KafkaRebalance goals match the goals configured in the default.goals section of the Kafka resource KafkaRebalance goals are not specified Otherwise, Cruise Control generates a new optimization proposal based on KafkaRebalance goals . If new proposals are generated with each refresh, this can impact performance monitoring. 20.2.2. Setting up alerts for anomaly detection Cruise control's anomaly detector provides metrics data for conditions that block the generation of optimization goals, such as broker failures. If you want more visibility, you can use the metrics provided by the anomaly detector to set up alerts and send out notifications. You can set up Cruise Control's anomaly notifier to route alerts based on these metrics through a specified notification channel. Alternatively, you can set up Prometheus to scrape the metrics data provided by the anomaly detector and generate alerts. Prometheus Alertmanager can then route the alerts generated by Prometheus. The Cruise Control documentation provides information on AnomalyDetector metrics and the anomaly notifier. 20.3. Example metrics files You can find example Grafana dashboards and other metrics configuration files in the example configuration files provided by AMQ Streams. Example metrics files provided with AMQ Streams 1 Example Grafana dashboards for the different AMQ Streams components. 2 Installation file for the Grafana image. 3 Additional configuration to scrape metrics for CPU, memory and disk volume usage, which comes directly from the OpenShift cAdvisor agent and kubelet on the nodes. 4 Hook definitions for sending notifications through Alertmanager. 5 Resources for deploying and configuring Alertmanager. 6 Alerting rules examples for use with Prometheus Alertmanager (deployed with Prometheus). 7 Installation resource file for the Prometheus image. 8 PodMonitor definitions translated by the Prometheus Operator into jobs for the Prometheus server to be able to scrape metrics data directly from pods. 9 Kafka Bridge resource with metrics enabled. 10 Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Kafka Connect. 11 Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Cruise Control. 12 Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Kafka and ZooKeeper. 13 Metrics configuration that defines Prometheus JMX Exporter relabeling rules for Kafka Mirror Maker 2.0. 20.3.1. Example Prometheus metrics configuration AMQ Streams uses the Prometheus JMX Exporter to expose metrics through an HTTP endpoint, which can be scraped by the Prometheus server. Grafana dashboards are dependent on Prometheus JMX Exporter relabeling rules, which are defined for AMQ Streams components in the custom resource configuration. A label is a name-value pair. Relabeling is the process of writing a label dynamically. For example, the value of a label may be derived from the name of a Kafka server and client ID. AMQ Streams provides example custom resource configuration YAML files with relabeling rules. When deploying Prometheus metrics configuration, you can can deploy the example custom resource or copy the metrics configuration to your own custom resource definition. Table 20.1. Example custom resources with metrics configuration Component Custom resource Example YAML file Kafka and ZooKeeper Kafka kafka-metrics.yaml Kafka Connect KafkaConnect kafka-connect-metrics.yaml Kafka MirrorMaker 2 KafkaMirrorMaker2 kafka-mirror-maker-2-metrics.yaml Kafka Bridge KafkaBridge kafka-bridge-metrics.yaml Cruise Control Kafka kafka-cruise-control-metrics.yaml 20.3.2. Example Prometheus rules for alert notifications Example Prometheus rules for alert notifications are provided with the example metrics configuration files provided by AMQ Streams. The rules are specified in the example prometheus-rules.yaml file for use in a Prometheus deployment . The prometheus-rules.yaml file contains example rules for the following components: Kafka ZooKeeper Entity Operator Kafka Connect Kafka Bridge MirrorMaker Kafka Exporter A description of each of the example rules is provided in the file. Alerting rules provide notifications about specific conditions observed in metrics. Rules are declared on the Prometheus server, but Prometheus Alertmanager is responsible for alert notifications. Prometheus alerting rules describe conditions using PromQL expressions that are continuously evaluated. When an alert expression becomes true, the condition is met and the Prometheus server sends alert data to the Alertmanager. Alertmanager then sends out a notification using the communication method configured for its deployment. General points about the alerting rule definitions: A for property is used with the rules to determine the period of time a condition must persist before an alert is triggered. A tick is a basic ZooKeeper time unit, which is measured in milliseconds and configured using the tickTime parameter of Kafka.spec.zookeeper.config . For example, if ZooKeeper tickTime=3000 , 3 ticks (3 x 3000) equals 9000 milliseconds. The availability of the ZookeeperRunningOutOfSpace metric and alert is dependent on the OpenShift configuration and storage implementation used. Storage implementations for certain platforms may not be able to supply the information on available space required for the metric to provide an alert. Alertmanager can be configured to use email, chat messages or other notification methods. Adapt the default configuration of the example rules according to your specific needs. 20.3.3. Example Grafana dashboards If you deploy Prometheus to provide metrics, you can use the example Grafana dashboards provided with AMQ Streams to monitor AMQ Streams components. Example dashboards are provided in the examples/metrics/grafana-dashboards directory as JSON files. All dashboards provide JVM metrics, as well as metrics specific to the component. For example, the Grafana dashboard for AMQ Streams operators provides information on the number of reconciliations or custom resources they are processing. The example dashboards don't show all the metrics supported by Kafka. The dashboards are populated with a representative set of metrics for monitoring. Table 20.2. Example Grafana dashboard files Component Example JSON file AMQ Streams operators strimzi-operators.json Kafka strimzi-kafka.json ZooKeeper strimzi-zookeeper.json Kafka Connect strimzi-kafka-connect.json Kafka MirrorMaker 2 strimzi-kafka-mirror-maker-2.json Kafka Bridge strimzi-kafka-bridge.json Cruise Control strimzi-cruise-control.json Kafka Exporter strimzi-kafka-exporter.json Note When metrics are not available to the Kafka Exporter, because there is no traffic in the cluster yet, the Kafka Exporter Grafana dashboard will show N/A for numeric fields and No data to show for graphs. 20.4. Enabling Prometheus metrics through configuration To enable and expose metrics in AMQ Streams for Prometheus, use metrics configuration properties. The following components require metricsConfig configuration to expose metrics: Kafka KafkaConnect MirrorMaker Cruise Control ZooKeeper This configuration enables the Prometheus JMX Exporter to expose metrics through an HTTP endpoint. The port for the JMX exporter HTTP endpoint is 9404. Prometheus scrapes this endpoint to collect Kafka metrics. You set the enableMetrics property to true in order to expose metrics for these components: Kafka Bridge OAuth 2.0 authentication and authorization framework Open Policy Agent (OPA) for authorization To deploy Prometheus metrics configuration in AMQ Streams, you can use your own configuration or the example custom resource configuration files provided with AMQ Streams: kafka-metrics.yaml kafka-connect-metrics.yaml kafka-mirror-maker-2-metrics.yaml kafka-bridge-metrics.yaml kafka-cruise-control-metrics.yaml oauth-metrics.yaml These files contain the necessary relabeling rules and configuration to enable Prometheus metrics. They are a good starting point for trying Prometheus with AMQ Streams. This procedure shows how to deploy example Prometheus metrics configuration in the Kafka resource. The process is the same when deploying the example files for other resources. If you wish to include Kafka Exporter metrics, add kafkaExporter configuration to your Kafka resource. Important Kafka Exporter only provides additional metrics related to consumer lag and consumer offsets. For regular Kafka metrics, you must configure the Prometheus metrics in Kafka brokers . Procedure Deploy the example custom resource with the Prometheus configuration. For example, for each Kafka resource you can apply the kafka-metrics.yaml file. Deploying the example configuration oc apply -f kafka-metrics.yaml Alternatively, you can copy the example configuration in kafka-metrics.yaml to your own Kafka resource. Copying the example configuration oc edit kafka <kafka_configuration_file> Copy the metricsConfig property and the ConfigMap it references to your Kafka resource. Example metrics configuration for Kafka apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # ... metricsConfig: 1 type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: kafka-metrics-config.yml --- kind: ConfigMap 2 apiVersion: v1 metadata: name: kafka-metrics labels: app: strimzi data: kafka-metrics-config.yml: | # metrics configuration... 1 Copy the metricsConfig property that references the ConfigMap that contains metrics configuration. 2 Copy the whole ConfigMap that specifies the metrics configuration. To deploy Kafka Exporter, add kafkaExporter configuration. kafkaExporter configuration is only specified in the Kafka resource. Example configuration for deploying Kafka Exporter apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: # ... kafkaExporter: image: my-registry.io/my-org/my-exporter-cluster:latest 1 groupRegex: ".*" 2 topicRegex: ".*" 3 groupExcludeRegex: "^excluded-.*" 4 topicExcludeRegex: "^excluded-.*" 5 resources: 6 requests: cpu: 200m memory: 64Mi limits: cpu: 500m memory: 128Mi logging: debug 7 enableSaramaLogging: true 8 template: 9 pod: metadata: labels: label1: value1 imagePullSecrets: - name: my-docker-credentials securityContext: runAsUser: 1000001 fsGroup: 0 terminationGracePeriodSeconds: 120 readinessProbe: 10 initialDelaySeconds: 15 timeoutSeconds: 5 livenessProbe: 11 initialDelaySeconds: 15 timeoutSeconds: 5 # ... 1 ADVANCED OPTION: Container image configuration, which is recommended only in special situations. 2 A regular expression to specify the consumer groups to include in the metrics. 3 A regular expression to specify the topics to include in the metrics. 4 A regular expression to specify the consumer groups to exclude in the metrics. 5 A regular expression to specify the topics to exclude in the metrics. 6 CPU and memory resources to reserve. 7 Logging configuration, to log messages with a given severity (debug, info, warn, error, fatal) or above. 8 Boolean to enable Sarama logging, a Go client library used by Kafka Exporter. 9 Customization of deployment templates and pods. 10 Healthcheck readiness probes. 11 Healthcheck liveness probes. Note For Kafka Exporter to be able to work properly, consumer groups need to be in use. Enabling metrics for Kafka Bridge To expose metrics for Kafka Bridge, set the enableMetrics property to true in the KafkaBridge resource. Example metrics configuration for Kafka Bridge apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaBridge metadata: name: my-bridge spec: # ... bootstrapServers: my-cluster-kafka:9092 http: # ... enableMetrics: true # ... Enabling metrics for OAuth 2.0 and OPA To expose metrics for OAuth 2.0 or OPA, set the enableMetrics property to true in the appropriate custom resource. OAuth 2.0 metrics Enable metrics for Kafka cluster authorization and Kafka listener authentication in the Kafka resource. You can also enable metrics for OAuth 2.0 authentication in the custom resource of other supported components . OPA metrics Enable metrics for Kafka cluster authorization the Kafka resource in the same way as for OAuth 2.0. In the following example, metrics are enabled for OAuth 2.0 listener authentication and OAuth 2.0 ( keycloak ) cluster authorization. Example cluster configuration with metrics enabled for OAuth 2.0 apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: myproject spec: kafka: # ... listeners: - name: external port: 9094 type: loadbalancer tls: true authentication: type: oauth enableMetrics: true configuration: #... authorization: type: keycloak enableMetrics: true # ... To use the OAuth 2.0 metrics with Prometheus, you can use the oauth-metrics.yaml file to deploy example Prometheus metrics configuration. Copy the ConfigMap configuration the oauth-metrics.yaml file contains to the same Kafka resource configuration file where you enabled metrics for OAuth 2.0. 20.5. Viewing Kafka metrics and dashboards in OpenShift When AMQ Streams is deployed to OpenShift Container Platform, metrics are provided through monitoring for user-defined projects . This OpenShift feature gives developers access to a separate Prometheus instance for monitoring their own projects (for example, a Kafka project). If monitoring for user-defined projects is enabled, the openshift-user-workload-monitoring project contains the following components: A Prometheus operator A Prometheus instance (automatically deployed by the Prometheus Operator) A Thanos Ruler instance AMQ Streams uses these components to consume metrics. A cluster administrator must enable monitoring for user-defined projects and then grant developers and other users permission to monitor applications within their own projects. Grafana deployment You can deploy a Grafana instance to the project containing your Kafka cluster. The example Grafana dashboards can then be used to visualize Prometheus metrics for AMQ Streams in the Grafana user interface. Important The openshift-monitoring project provides monitoring for core platform components. Do not use the Prometheus and Grafana components in this project to configure monitoring for AMQ Streams on OpenShift Container Platform 4.x. Procedure outline To set up AMQ Streams monitoring in OpenShift Container Platform, follow these procedures in order: Prerequisite: Deploy the Prometheus metrics configuration Deploy the Prometheus resources Create a service account for Grafana Deploy Grafana with a Prometheus datasource Create a Route to the Grafana Service Import the example Grafana dashboards 20.5.1. Prerequisites You have deployed the Prometheus metrics configuration using the example YAML files. Monitoring for user-defined projects is enabled. A cluster administrator has created a cluster-monitoring-config config map in your OpenShift cluster. A cluster administrator has assigned you a monitoring-rules-edit or monitoring-edit role. For more information on creating a cluster-monitoring-config config map and granting users permission to monitor user-defined projects, see the OpenShift documentation . 20.5.2. Deploying the Prometheus resources Use Prometheus to obtain monitoring data in your Kafka cluster. You can use your own Prometheus deployment or deploy Prometheus using the example metrics configuration files provided by AMQ Streams. To use the example files, you configure and deploy the PodMonitor resources. The PodMonitors scrape data directly from pods for Apache Kafka, ZooKeeper, Operators, the Kafka Bridge, and Cruise Control. You then deploy the example alerting rules for Alertmanager. Prerequisites A running Kafka cluster. Check the example alerting rules provided with AMQ Streams. Procedure Check that monitoring for user-defined projects is enabled: oc get pods -n openshift-user-workload-monitoring If enabled, pods for the monitoring components are returned. For example: NAME READY STATUS RESTARTS AGE prometheus-operator-5cc59f9bc6-kgcq8 1/1 Running 0 25s prometheus-user-workload-0 5/5 Running 1 14s prometheus-user-workload-1 5/5 Running 1 14s thanos-ruler-user-workload-0 3/3 Running 0 14s thanos-ruler-user-workload-1 3/3 Running 0 14s If no pods are returned, monitoring for user-defined projects is disabled. See the Prerequisites in Section 20.5, "Viewing Kafka metrics and dashboards in OpenShift" . Multiple PodMonitor resources are defined in examples/metrics/prometheus-install/strimzi-pod-monitor.yaml . For each PodMonitor resource, edit the spec.namespaceSelector.matchNames property: apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: cluster-operator-metrics labels: app: strimzi spec: selector: matchLabels: strimzi.io/kind: cluster-operator namespaceSelector: matchNames: - <project-name> 1 podMetricsEndpoints: - path: /metrics port: http # ... 1 The project where the pods to scrape the metrics from are running, for example, Kafka . Deploy the strimzi-pod-monitor.yaml file to the project where your Kafka cluster is running: oc apply -f strimzi-pod-monitor.yaml -n MY-PROJECT Deploy the example Prometheus rules to the same project: oc apply -f prometheus-rules.yaml -n MY-PROJECT 20.5.3. Creating a service account for Grafana A Grafana instance for AMQ Streams needs to run with a service account that is assigned the cluster-monitoring-view role. Create a service account if you are using Grafana to present metrics for monitoring. Prerequisites Deploy the Prometheus resources Procedure Create a ServiceAccount for Grafana in the project containing your Kafka cluster: oc create sa grafana-service-account -n my-project In this example, a service account named grafana-service-account is created in the my-project namespace. Create a ClusterRoleBinding resource that assigns the cluster-monitoring-view role to the Grafana ServiceAccount . Here the resource is named grafana-cluster-monitoring-binding . apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: grafana-cluster-monitoring-binding labels: app: strimzi subjects: - kind: ServiceAccount name: grafana-service-account namespace: my-project roleRef: kind: ClusterRole name: cluster-monitoring-view apiGroup: rbac.authorization.k8s.io Deploy the ClusterRoleBinding to the same project: oc apply -f grafana-cluster-monitoring-binding.yaml -n my-project Create a token secret for the service account: apiVersion: v1 kind: Secret metadata: name: secret-sa annotations: kubernetes.io/service-account.name: "grafana-service-account" 1 type: kubernetes.io/service-account-token 2 1 Specifies the service account. 2 Specifies a service account token secret. Create the Secret object and access token: oc create -f <secret_configuration>.yaml You need the access token when deploying Grafana. 20.5.4. Deploying Grafana with a Prometheus datasource Deploy Grafana to present Prometheus metrics. A Grafana application requires configuration for the OpenShift Container Platform monitoring stack. OpenShift Container Platform includes a Thanos Querier instance in the openshift-monitoring project. Thanos Querier is used to aggregate platform metrics. To consume the required platform metrics, your Grafana instance requires a Prometheus data source that can connect to Thanos Querier. To configure this connection, you create a config map that authenticates, by using a token, to the oauth-proxy sidecar that runs alongside Thanos Querier. A datasource.yaml file is used as the source of the config map. Finally, you deploy the Grafana application with the config map mounted as a volume to the project containing your Kafka cluster. Prerequisites You have deployed Prometheus resources . You have created a service account for Grafana . Procedure Get the access token of the Grafana ServiceAccount : oc describe sa/grafana-service-account | grep Tokens: oc describe secret grafana-service-account-token-mmlp9 | grep token: In this example, the service account is named grafana-service-account . Copy the access token to use in the step. Create a datasource.yaml file containing the Thanos Querier configuration for Grafana. Paste the access token into the httpHeaderValue1 property as indicated. apiVersion: 1 datasources: - name: Prometheus type: prometheus url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 access: proxy basicAuth: false withCredentials: false isDefault: true jsonData: timeInterval: 5s tlsSkipVerify: true httpHeaderName1: "Authorization" secureJsonData: httpHeaderValue1: "Bearer USD{ GRAFANA-ACCESS-TOKEN }" 1 editable: true 1 GRAFANA-ACCESS-TOKEN : The value of the access token for the Grafana ServiceAccount . Create a config map named grafana-config from the datasource.yaml file: oc create configmap grafana-config --from-file=datasource.yaml -n MY-PROJECT Create a Grafana application consisting of a Deployment and a Service . The grafana-config config map is mounted as a volume for the datasource configuration. apiVersion: apps/v1 kind: Deployment metadata: name: grafana labels: app: strimzi spec: replicas: 1 selector: matchLabels: name: grafana template: metadata: labels: name: grafana spec: serviceAccountName: grafana-service-account containers: - name: grafana image: grafana/grafana:10.0.3 ports: - name: grafana containerPort: 3000 protocol: TCP volumeMounts: - name: grafana-data mountPath: /var/lib/grafana - name: grafana-logs mountPath: /var/log/grafana - name: grafana-config mountPath: /etc/grafana/provisioning/datasources/datasource.yaml readOnly: true subPath: datasource.yaml readinessProbe: httpGet: path: /api/health port: 3000 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /api/health port: 3000 initialDelaySeconds: 15 periodSeconds: 20 volumes: - name: grafana-data emptyDir: {} - name: grafana-logs emptyDir: {} - name: grafana-config configMap: name: grafana-config --- apiVersion: v1 kind: Service metadata: name: grafana labels: app: strimzi spec: ports: - name: grafana port: 3000 targetPort: 3000 protocol: TCP selector: name: grafana type: ClusterIP Deploy the Grafana application to the project containing your Kafka cluster: oc apply -f <grafana-application> -n <my-project> 20.5.5. Creating a route to the Grafana Service You can access the Grafana user interface through a Route that exposes the Grafana service. Prerequisites Deploy the Prometheus resources Create a service account for Grafana Deploy Grafana with a Prometheus datasource Procedure Create an edge route to the grafana service: oc create route edge <my-grafana-route> --service=grafana --namespace= KAFKA-NAMESPACE 20.5.6. Importing the example Grafana dashboards Use Grafana to provide visualizations of Prometheus metrics on customizable dashboards. AMQ Streams provides example dashboard configuration files for Grafana in JSON format. examples/metrics/grafana-dashboards This procedure uses the example Grafana dashboards. The example dashboards are a good starting point for monitoring key metrics, but they don't show all the metrics supported by Kafka. You can modify the example dashboards or add other metrics, depending on your infrastructure. Prerequisites Deploy the Prometheus resources Create a service account for Grafana Deploy Grafana with a Prometheus datasource Create a Route to the Grafana Service Procedure Get the details of the Route to the Grafana Service. For example: oc get routes NAME HOST/PORT PATH SERVICES MY-GRAFANA-ROUTE MY-GRAFANA-ROUTE-amq-streams.net grafana In a web browser, access the Grafana login screen using the URL for the Route host and port. Enter your user name and password, and then click Log In . The default Grafana user name and password are both admin . After logging in for the first time, you can change the password. In Configuration > Data Sources , check that the Prometheus data source was created. The data source was created in Section 20.5.4, "Deploying Grafana with a Prometheus datasource" . Click the + icon and then click Import . In examples/metrics/grafana-dashboards , copy the JSON of the dashboard to import. Paste the JSON into the text box, and then click Load . Repeat steps 5-7 for the other example Grafana dashboards. The imported Grafana dashboards are available to view from the Dashboards home page.
metrics ├── grafana-dashboards 1 │ ├── strimzi-cruise-control.json │ ├── strimzi-kafka-bridge.json │ ├── strimzi-kafka-connect.json │ ├── strimzi-kafka-exporter.json │ ├── strimzi-kafka-mirror-maker-2.json │ ├── strimzi-kafka.json │ ├── strimzi-operators.json │ └── strimzi-zookeeper.json ├── grafana-install │ └── grafana.yaml 2 ├── prometheus-additional-properties │ └── prometheus-additional.yaml 3 ├── prometheus-alertmanager-config │ └── alert-manager-config.yaml 4 ├── prometheus-install │ ├── alert-manager.yaml 5 │ ├── prometheus-rules.yaml 6 │ ├── prometheus.yaml 7 │ └── strimzi-pod-monitor.yaml 8 ├── kafka-bridge-metrics.yaml 9 ├── kafka-connect-metrics.yaml 10 ├── kafka-cruise-control-metrics.yaml 11 ├── kafka-metrics.yaml 12 └── kafka-mirror-maker-2-metrics.yaml 13 apply -f kafka-metrics.yaml edit kafka <kafka_configuration_file> apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # metricsConfig: 1 type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: kafka-metrics-config.yml --- kind: ConfigMap 2 apiVersion: v1 metadata: name: kafka-metrics labels: app: strimzi data: kafka-metrics-config.yml: | # metrics configuration apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: # kafkaExporter: image: my-registry.io/my-org/my-exporter-cluster:latest 1 groupRegex: ".*" 2 topicRegex: ".*" 3 groupExcludeRegex: "^excluded-.*" 4 topicExcludeRegex: "^excluded-.*" 5 resources: 6 requests: cpu: 200m memory: 64Mi limits: cpu: 500m memory: 128Mi logging: debug 7 enableSaramaLogging: true 8 template: 9 pod: metadata: labels: label1: value1 imagePullSecrets: - name: my-docker-credentials securityContext: runAsUser: 1000001 fsGroup: 0 terminationGracePeriodSeconds: 120 readinessProbe: 10 initialDelaySeconds: 15 timeoutSeconds: 5 livenessProbe: 11 initialDelaySeconds: 15 timeoutSeconds: 5 apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaBridge metadata: name: my-bridge spec: # bootstrapServers: my-cluster-kafka:9092 http: # enableMetrics: true # apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster namespace: myproject spec: kafka: # listeners: - name: external port: 9094 type: loadbalancer tls: true authentication: type: oauth enableMetrics: true configuration: # authorization: type: keycloak enableMetrics: true # get pods -n openshift-user-workload-monitoring NAME READY STATUS RESTARTS AGE prometheus-operator-5cc59f9bc6-kgcq8 1/1 Running 0 25s prometheus-user-workload-0 5/5 Running 1 14s prometheus-user-workload-1 5/5 Running 1 14s thanos-ruler-user-workload-0 3/3 Running 0 14s thanos-ruler-user-workload-1 3/3 Running 0 14s apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: cluster-operator-metrics labels: app: strimzi spec: selector: matchLabels: strimzi.io/kind: cluster-operator namespaceSelector: matchNames: - <project-name> 1 podMetricsEndpoints: - path: /metrics port: http apply -f strimzi-pod-monitor.yaml -n MY-PROJECT apply -f prometheus-rules.yaml -n MY-PROJECT create sa grafana-service-account -n my-project apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: grafana-cluster-monitoring-binding labels: app: strimzi subjects: - kind: ServiceAccount name: grafana-service-account namespace: my-project roleRef: kind: ClusterRole name: cluster-monitoring-view apiGroup: rbac.authorization.k8s.io apply -f grafana-cluster-monitoring-binding.yaml -n my-project apiVersion: v1 kind: Secret metadata: name: secret-sa annotations: kubernetes.io/service-account.name: "grafana-service-account" 1 type: kubernetes.io/service-account-token 2 create -f <secret_configuration>.yaml describe sa/grafana-service-account | grep Tokens: describe secret grafana-service-account-token-mmlp9 | grep token: apiVersion: 1 datasources: - name: Prometheus type: prometheus url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091 access: proxy basicAuth: false withCredentials: false isDefault: true jsonData: timeInterval: 5s tlsSkipVerify: true httpHeaderName1: "Authorization" secureJsonData: httpHeaderValue1: "Bearer USD{ GRAFANA-ACCESS-TOKEN }" 1 editable: true create configmap grafana-config --from-file=datasource.yaml -n MY-PROJECT apiVersion: apps/v1 kind: Deployment metadata: name: grafana labels: app: strimzi spec: replicas: 1 selector: matchLabels: name: grafana template: metadata: labels: name: grafana spec: serviceAccountName: grafana-service-account containers: - name: grafana image: grafana/grafana:10.0.3 ports: - name: grafana containerPort: 3000 protocol: TCP volumeMounts: - name: grafana-data mountPath: /var/lib/grafana - name: grafana-logs mountPath: /var/log/grafana - name: grafana-config mountPath: /etc/grafana/provisioning/datasources/datasource.yaml readOnly: true subPath: datasource.yaml readinessProbe: httpGet: path: /api/health port: 3000 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /api/health port: 3000 initialDelaySeconds: 15 periodSeconds: 20 volumes: - name: grafana-data emptyDir: {} - name: grafana-logs emptyDir: {} - name: grafana-config configMap: name: grafana-config --- apiVersion: v1 kind: Service metadata: name: grafana labels: app: strimzi spec: ports: - name: grafana port: 3000 targetPort: 3000 protocol: TCP selector: name: grafana type: ClusterIP apply -f <grafana-application> -n <my-project> create route edge <my-grafana-route> --service=grafana --namespace= KAFKA-NAMESPACE get routes NAME HOST/PORT PATH SERVICES MY-GRAFANA-ROUTE MY-GRAFANA-ROUTE-amq-streams.net grafana