Chapter 3. Cluster logging
Chapter 3. Cluster logging 3.1. Using OpenShift Logging with OpenShift Serverless 3.1.1. About deploying the logging subsystem for Red Hat OpenShift OpenShift Container Platform cluster administrators can deploy the logging subsystem using the OpenShift Container Platform web console or CLI to install the OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator. When the Operators are installed, you create a ClusterLogging custom resource (CR) to schedule logging subsystem pods and other resources necessary to support the logging subsystem. The Operators are responsible for deploying, upgrading, and maintaining the logging subsystem. The ClusterLogging CR defines a complete logging subsystem environment that includes all the components of the logging stack to collect, store and visualize logs. The Red Hat OpenShift Logging Operator watches the logging subsystem CR and adjusts the logging deployment accordingly. Administrators and application developers can view the logs of the projects for which they have view access. 3.1.2. About deploying and configuring the logging subsystem for Red Hat OpenShift The logging subsystem is designed to be used with the default configuration, which is tuned for small to medium sized OpenShift Container Platform clusters. The installation instructions that follow include a sample ClusterLogging custom resource (CR), which you can use to create a logging subsystem instance and configure your logging subsystem environment. If you want to use the default logging subsystem install, you can use the sample CR directly. If you want to customize your deployment, make changes to the sample CR as needed. The following describes the configurations you can make when installing your OpenShift Logging instance or modify after installation. See the Configuring sections for more information on working with each component, including modifications you can make outside of the ClusterLogging custom resource. 3.1.2.1. Configuring and Tuning the logging subsystem You can configure your logging subsystem by modifying the ClusterLogging custom resource deployed in the openshift-logging project. You can modify any of the following components upon install or after install: Memory and CPU You can adjust both the CPU and memory limits for each component by modifying the resources block with valid memory and CPU values: spec: logStore: elasticsearch: resources: limits: cpu: memory: 16Gi requests: cpu: 500m memory: 16Gi type: "elasticsearch" collection: logs: fluentd: resources: limits: cpu: memory: requests: cpu: memory: type: "fluentd" visualization: kibana: resources: limits: cpu: memory: requests: cpu: memory: type: kibana Elasticsearch storage You can configure a persistent storage class and size for the Elasticsearch cluster using the storageClass name and size parameters. The Red Hat OpenShift Logging Operator creates a persistent volume claim (PVC) for each data node in the Elasticsearch cluster based on these parameters. spec: logStore: type: "elasticsearch" elasticsearch: nodeCount: 3 storage: storageClassName: "gp2" size: "200G" This example specifies each data node in the cluster will be bound to a PVC that requests "200G" of "gp2" storage. Each primary shard will be backed by a single replica. Note Omitting the storage block results in a deployment that includes ephemeral storage only. spec: logStore: type: "elasticsearch" elasticsearch: nodeCount: 3 storage: {} Elasticsearch replication policy You can set the policy that defines how Elasticsearch shards are replicated across data nodes in the cluster: FullRedundancy . The shards for each index are fully replicated to every data node. MultipleRedundancy . The shards for each index are spread over half of the data nodes. SingleRedundancy . A single copy of each shard. Logs are always available and recoverable as long as at least two data nodes exist. ZeroRedundancy . No copies of any shards. Logs may be unavailable (or lost) in the event a node is down or fails. 3.1.2.2. Sample modified ClusterLogging custom resource The following is an example of a ClusterLogging custom resource modified using the options previously described. Sample modified ClusterLogging custom resource apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: "openshift-logging" spec: managementState: "Managed" logStore: type: "elasticsearch" retentionPolicy: application: maxAge: 1d infra: maxAge: 7d audit: maxAge: 7d elasticsearch: nodeCount: 3 resources: limits: cpu: 200m memory: 16Gi requests: cpu: 200m memory: 16Gi storage: storageClassName: "gp2" size: "200G" redundancyPolicy: "SingleRedundancy" visualization: type: "kibana" kibana: resources: limits: memory: 1Gi requests: cpu: 500m memory: 1Gi replicas: 1 collection: logs: type: "fluentd" fluentd: resources: limits: memory: 1Gi requests: cpu: 200m memory: 1Gi 3.2. Finding logs for Knative Serving components You can find the logs for Knative Serving components using the following procedure. 3.2.1. Using OpenShift Logging to find logs for Knative Serving components Prerequisites Install the OpenShift CLI ( oc ). Procedure Get the Kibana route: USD oc -n openshift-logging get route kibana Use the route's URL to navigate to the Kibana dashboard and log in. Check that the index is set to .all . If the index is not set to .all , only the OpenShift Container Platform system logs will be listed. Filter the logs by using the knative-serving namespace. Enter kubernetes.namespace_name:knative-serving in the search box to filter results. Note Knative Serving uses structured logging by default. You can enable the parsing of these logs by customizing the OpenShift Logging Fluentd settings. This makes the logs more searchable and enables filtering on the log level to quickly identify issues. 3.3. Finding logs for Knative Serving services You can find the logs for Knative Serving services using the following procedure. 3.3.1. Using OpenShift Logging to find logs for services deployed with Knative Serving With OpenShift Logging, the logs that your applications write to the console are collected in Elasticsearch. The following procedure outlines how to apply these capabilities to applications deployed by using Knative Serving. Prerequisites Install the OpenShift CLI ( oc ). Procedure Get the Kibana route: USD oc -n openshift-logging get route kibana Use the route's URL to navigate to the Kibana dashboard and log in. Check that the index is set to .all . If the index is not set to .all , only the OpenShift system logs will be listed. Filter the logs by using the knative-serving namespace. Enter a filter for the service in the search box to filter results. Example filter kubernetes.namespace_name:default AND kubernetes.labels.serving_knative_dev\/service:{service_name} You can also filter by using /configuration or /revision . Narrow your search by using kubernetes.container_name:<user_container> to only display the logs generated by your application. Otherwise, you will see logs from the queue-proxy. Note Use JSON-based structured logging in your application to allow for the quick filtering of these logs in production environments.
spec: logStore: elasticsearch: resources: limits: cpu: memory: 16Gi requests: cpu: 500m memory: 16Gi type: "elasticsearch" collection: logs: fluentd: resources: limits: cpu: memory: requests: cpu: memory: type: "fluentd" visualization: kibana: resources: limits: cpu: memory: requests: cpu: memory: type: kibana spec: logStore: type: "elasticsearch" elasticsearch: nodeCount: 3 storage: storageClassName: "gp2" size: "200G" spec: logStore: type: "elasticsearch" elasticsearch: nodeCount: 3 storage: {} apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: "openshift-logging" spec: managementState: "Managed" logStore: type: "elasticsearch" retentionPolicy: application: maxAge: 1d infra: maxAge: 7d audit: maxAge: 7d elasticsearch: nodeCount: 3 resources: limits: cpu: 200m memory: 16Gi requests: cpu: 200m memory: 16Gi storage: storageClassName: "gp2" size: "200G" redundancyPolicy: "SingleRedundancy" visualization: type: "kibana" kibana: resources: limits: memory: 1Gi requests: cpu: 500m memory: 1Gi replicas: 1 collection: logs: type: "fluentd" fluentd: resources: limits: memory: 1Gi requests: cpu: 200m memory: 1Gi oc -n openshift-logging get route kibana oc -n openshift-logging get route kibana kubernetes.namespace_name:default AND kubernetes.labels.serving_knative_dev\/service:{service_name}