Chapter 5. Installing Logging
Chapter 5. Installing Logging You can deploy logging by installing the Red Hat OpenShift Logging Operator. The Red Hat OpenShift Logging Operator creates and manages the components of the logging stack. Note Logging is provided as an installable component, with a distinct release cycle from the core OpenShift Container Platform. The Red Hat OpenShift Container Platform Life Cycle Policy outlines release compatibility. Important For new installations, use Vector and LokiStack. Elasticsearch and Fluentd are deprecated and are planned to be removed in a future release. 5.1. Installing the Red Hat OpenShift Logging Operator by using the web console You can install the Red Hat OpenShift Logging Operator by using the OpenShift Container Platform web console. Prerequisites You have administrator permissions. You have access to the OpenShift Container Platform web console. Procedure In the OpenShift Container Platform web console, click Operators OperatorHub . Type OpenShift Logging in the Filter by keyword box. Choose Red Hat OpenShift Logging from the list of available Operators, and click Install . Ensure that A specific namespace on the cluster is selected under Installation mode . Ensure that Operator recommended namespace is openshift-logging under Installed Namespace . Select Enable operator recommended cluster monitoring on this namespace . This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-logging namespace. Select stable-5.y as the Update channel . Note The stable channel only provides updates to the most recent release of logging. To continue receiving updates for prior releases, you must change your subscription channel to stable-x.y , where x.y represents the major and minor version of logging you have installed. For example, stable-5.7 . Select an Update approval . The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available. The Manual strategy requires a user with appropriate credentials to approve the Operator update. Select Enable or Disable for the Console plugin . Click Install . Verification Verify that the Red Hat OpenShift Logging Operator is installed by switching to the Operators Installed Operators page. In the Status column, verify that you see green checkmarks with InstallSucceeded and the text Up to date . Important An Operator might display a Failed status before the installation finishes. If the Operator install completes with an InstallSucceeded message, refresh the page. If the Operator does not show as installed, choose one of the following troubleshooting options: Go to the Operators Installed Operators page, and inspect the Status column for any errors or failures. Go to the Workloads Pods page, and check the logs in any pods in the openshift-logging project that are reporting issues. 5.2. Creating a ClusterLogging object by using the web console After you have installed the logging Operators, you must create a ClusterLogging custom resource to configure log storage, visualization, and the log collector for your cluster. Prerequisites You have installed the Red Hat OpenShift Logging Operator. You have access to the OpenShift Container Platform web console Administrator perspective. Procedure Navigate to the Custom Resource Definitions page. On the Custom Resource Definitions page, click ClusterLogging . On the Custom Resource Definition details page, select View Instances from the Actions menu. On the ClusterLoggings page, click Create ClusterLogging . In the collection section, select a Collector Implementation . Note Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead. In the logStore section, select a type. Note The OpenShift Elasticsearch Operator is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. Click Create . 5.3. Installing the Red Hat OpenShift Logging Operator by using the CLI You can use the OpenShift CLI ( oc ) to install the Red Hat OpenShift Logging Operator. Prerequisites You have administrator permissions. You have installed the OpenShift CLI ( oc ). Procedure Create a Namespace object as a YAML file: Example Namespace object apiVersion: v1 kind: Namespace metadata: name: openshift-operators-redhat 1 annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-monitoring: "true" 2 1 You must specify the openshift-operators-redhat namespace. To prevent possible conflicts with metrics, you should configure the Prometheus Cluster Monitoring stack to scrape metrics from the openshift-operators-redhat namespace and not the openshift-operators namespace. The openshift-operators namespace might contain community Operators, which are untrusted and could publish a metric with the same name as an OpenShift Container Platform metric, which would cause conflicts. 2 String. You must specify this label as shown to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace. Apply the Namespace object by running the following command: USD oc apply -f <filename>.yaml Create a Namespace object for the Red Hat OpenShift Logging Operator: Example Namespace object apiVersion: v1 kind: Namespace metadata: name: openshift-logging annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-monitoring: "true" Apply the Namespace object by running the following command: USD oc apply -f <filename>.yaml Create an OperatorGroup object as a YAML file: Example OperatorGroup object apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: cluster-logging namespace: openshift-logging 1 spec: targetNamespaces: - openshift-logging 2 1 2 You must specify openshift-logging as the name of the namespace. Apply the OperatorGroup object by running the following command: USD oc apply -f <filename>.yaml Create a Subscription object to subscribe the namespace to the Red Hat OpenShift Logging Operator: Example Subscription object apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-logging namespace: openshift-logging 1 spec: channel: stable 2 name: cluster-logging source: redhat-operators 3 sourceNamespace: openshift-marketplace 1 You must specify openshift-logging as the name of the namespace. 2 Specify stable or stable-x.y as the channel. 3 Specify redhat-operators . If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator Lifecycle Manager (OLM). Apply the subscription by running the following command: USD oc apply -f <filename>.yaml The Red Hat OpenShift Logging Operator is installed to the openshift-logging namespace. Verification Run the following command: USD oc get csv -n <namespace> Observe the output and confirm that the Red Hat OpenShift Logging Operator exists in the namespace: Example output NAMESPACE NAME DISPLAY VERSION REPLACES PHASE ... openshift-logging clusterlogging.5.7.0-202007012112.p0 OpenShift Logging 5.7.0-202007012112.p0 Succeeded ... 5.4. Creating a ClusterLogging object by using the CLI This default logging configuration supports a wide array of environments. Review the topics on tuning and configuring components for information about modifications you can make. Prerequisites You have installed the Red Hat OpenShift Logging Operator. You have installed the OpenShift Elasticsearch Operator for your log store. You have installed the OpenShift CLI ( oc ). Procedure Create a ClusterLogging object as a YAML file: Example ClusterLogging object apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging spec: managementState: Managed 2 logStore: type: elasticsearch 3 retentionPolicy: 4 application: maxAge: 1d infra: maxAge: 7d audit: maxAge: 7d elasticsearch: nodeCount: 3 5 storage: storageClassName: <storage_class_name> 6 size: 200G resources: 7 limits: memory: 16Gi requests: memory: 16Gi proxy: 8 resources: limits: memory: 256Mi requests: memory: 256Mi redundancyPolicy: SingleRedundancy visualization: type: kibana 9 kibana: replicas: 1 collection: type: fluentd 10 fluentd: {} 1 The name must be instance . 2 The OpenShift Logging management state. In some cases, if you change the OpenShift Logging defaults, you must set this to Unmanaged . However, an unmanaged deployment does not receive updates until OpenShift Logging is placed back into a managed state. 3 Settings for configuring Elasticsearch. Using the CR, you can configure shard replication policy and persistent storage. 4 Specify the length of time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 7d for seven days. Logs older than the maxAge are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source. 5 Specify the number of Elasticsearch nodes. See the note that follows this list. 6 Enter the name of an existing storage class for Elasticsearch storage. For best performance, specify a storage class that allocates block storage. If you do not specify a storage class, OpenShift Logging uses ephemeral storage. 7 Specify the CPU and memory requests for Elasticsearch as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 16Gi for the memory request and 1 for the CPU request. 8 Specify the CPU and memory requests for the Elasticsearch proxy as needed. If you leave these values blank, the OpenShift Elasticsearch Operator sets default values that should be sufficient for most deployments. The default values are 256Mi for the memory request and 100m for the CPU request. 9 Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes. For more information, see Configuring the log visualizer . 10 Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see "Configuring Fluentd". Note The maximum number of Elasticsearch control plane nodes is three. If you specify a nodeCount greater than 3 , OpenShift Container Platform creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as data-only nodes, using client and data roles. Control plane nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded. For example, if nodeCount=4 , the following nodes are created: USD oc get deployment Example output NAME READY UP-TO-DATE AVAILABLE AGE cluster-logging-operator 1/1 1 1 18h elasticsearch-cd-x6kdekli-1 1/1 1 1 6m54s elasticsearch-cdm-x6kdekli-1 1/1 1 1 18h elasticsearch-cdm-x6kdekli-2 1/1 1 1 6m49s elasticsearch-cdm-x6kdekli-3 1/1 1 1 6m44s The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes. Verification You can verify the installation by listing the pods in the openshift-logging project. List the pods by running the following command: USD oc get pods -n openshift-logging Observe the pods for components of the logging, similar to the following list: Example output NAME READY STATUS RESTARTS AGE cluster-logging-operator-66f77ffccb-ppzbg 1/1 Running 0 7m elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp 2/2 Running 0 2m40s elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc 2/2 Running 0 2m36s elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2 2/2 Running 0 2m4s collector-587vb 1/1 Running 0 2m26s collector-7mpb9 1/1 Running 0 2m30s collector-flm6j 1/1 Running 0 2m33s collector-gn4rn 1/1 Running 0 2m26s collector-nlgb6 1/1 Running 0 2m30s collector-snpkt 1/1 Running 0 2m28s kibana-d6d5668c5-rppqm 2/2 Running 0 2m39s 5.5. Postinstallation tasks After you have installed the Red Hat OpenShift Logging Operator, you can configure your deployment by creating and modifying a ClusterLogging custom resource (CR). Tip If you are not using the Elasticsearch log store, you can remove the internal Elasticsearch logStore and Kibana visualization components from the ClusterLogging custom resource (CR). Removing these components is optional but saves resources. See Removing unused components if you do not use the Elasticsearch log store . 5.5.1. About the ClusterLogging custom resource To make changes to your logging environment, create and modify the ClusterLogging custom resource (CR). Sample ClusterLogging custom resource (CR) apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging 2 spec: managementState: Managed 3 # ... 1 The CR name must be instance . 2 The CR must be installed to the openshift-logging namespace. 3 The Red Hat OpenShift Logging Operator management state. When the state is set to unmanaged , the Operator is in an unsupported state and does not receive updates. 5.5.2. Configuring log storage You can configure which log storage type your logging uses by modifying the ClusterLogging custom resource (CR). Prerequisites You have administrator permissions. You have installed the OpenShift CLI ( oc ). You have installed the Red Hat OpenShift Logging Operator and an internal log store that is either the LokiStack or Elasticsearch. You have created a ClusterLogging CR. Note The OpenShift Elasticsearch Operator is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to using the OpenShift Elasticsearch Operator to manage the default log storage, you can use the Loki Operator. Procedure Modify the ClusterLogging CR logStore spec: ClusterLogging CR example apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: # ... spec: # ... logStore: type: <log_store_type> 1 elasticsearch: 2 nodeCount: <integer> resources: {} storage: {} redundancyPolicy: <redundancy_type> 3 lokistack: 4 name: {} # ... 1 Specify the log store type. This can be either lokistack or elasticsearch . 2 Optional configuration options for the Elasticsearch log store. 3 Specify the redundancy type. This value can be ZeroRedundancy , SingleRedundancy , MultipleRedundancy , or FullRedundancy . 4 Optional configuration options for LokiStack. Example ClusterLogging CR to specify LokiStack as the log store apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging spec: managementState: Managed logStore: type: lokistack lokistack: name: logging-loki # ... Apply the ClusterLogging CR by running the following command: USD oc apply -f <filename>.yaml 5.5.3. Configuring the log collector You can configure which log collector type your logging uses by modifying the ClusterLogging custom resource (CR). Note Fluentd is deprecated and is planned to be removed in a future release. Red Hat provides bug fixes and support for this feature during the current release lifecycle, but this feature no longer receives enhancements. As an alternative to Fluentd, you can use Vector instead. Prerequisites You have administrator permissions. You have installed the OpenShift CLI ( oc ). You have installed the Red Hat OpenShift Logging Operator. You have created a ClusterLogging CR. Procedure Modify the ClusterLogging CR collection spec: ClusterLogging CR example apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: # ... spec: # ... collection: type: <log_collector_type> 1 resources: {} tolerations: {} # ... 1 The log collector type you want to use for the logging. This can be vector or fluentd . Apply the ClusterLogging CR by running the following command: USD oc apply -f <filename>.yaml 5.5.4. Configuring the log visualizer You can configure which log visualizer type your logging uses by modifying the ClusterLogging custom resource (CR). Prerequisites You have administrator permissions. You have installed the OpenShift CLI ( oc ). You have installed the Red Hat OpenShift Logging Operator. You have created a ClusterLogging CR. Important If you want to use the OpenShift Container Platform web console for visualization, you must enable the logging Console plugin. See the documentation about "Log visualization with the web console". Procedure Modify the ClusterLogging CR visualization spec: ClusterLogging CR example apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: # ... spec: # ... visualization: type: <visualizer_type> 1 kibana: 2 resources: {} nodeSelector: {} proxy: {} replicas: {} tolerations: {} ocpConsole: 3 logsLimit: {} timeout: {} # ... 1 The type of visualizer you want to use for your logging. This can be either kibana or ocp-console . The Kibana console is only compatible with deployments that use Elasticsearch log storage, while the OpenShift Container Platform console is only compatible with LokiStack deployments. 2 Optional configurations for the Kibana console. 3 Optional configurations for the OpenShift Container Platform web console. Apply the ClusterLogging CR by running the following command: USD oc apply -f <filename>.yaml 5.5.5. Allowing traffic between projects when network isolation is enabled Your cluster network provider might enforce network isolation. If so, you must allow network traffic between the projects that contain the operators deployed by OpenShift Logging. Network isolation blocks network traffic between pods or services that are in different projects. The logging installs the OpenShift Elasticsearch Operator in the openshift-operators-redhat project and the Red Hat OpenShift Logging Operator in the openshift-logging project. Therefore, you must allow traffic between these two projects. OpenShift Container Platform offers two supported choices for the default Container Network Interface (CNI) network provider, OpenShift SDN and OVN-Kubernetes. These two providers implement various network isolation policies. OpenShift SDN has three modes: network policy This is the default mode. If no policy is defined, it allows all traffic. However, if a user defines a policy, they typically start by denying all traffic and then adding exceptions. This process might break applications that are running in different projects. Therefore, explicitly configure the policy to allow traffic to egress from one logging-related project to the other. multitenant This mode enforces network isolation. You must join the two logging-related projects to allow traffic between them. subnet This mode allows all traffic. It does not enforce network isolation. No action is needed. OVN-Kubernetes always uses a network policy . Therefore, as with OpenShift SDN, you must configure the policy to allow traffic to egress from one logging-related project to the other. Procedure If you are using OpenShift SDN in multitenant mode, join the two projects. For example: USD oc adm pod-network join-projects --to=openshift-operators-redhat openshift-logging Otherwise, for OpenShift SDN in network policy mode and OVN-Kubernetes, perform the following actions: Set a label on the openshift-operators-redhat namespace. For example: USD oc label namespace openshift-operators-redhat project=openshift-operators-redhat Create a network policy object in the openshift-logging namespace that allows ingress from the openshift-operators-redhat , openshift-monitoring and openshift-ingress projects to the openshift-logging project. For example: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-from-openshift-monitoring-ingress-operators-redhat spec: ingress: - from: - podSelector: {} - from: - namespaceSelector: matchLabels: project: "openshift-operators-redhat" - from: - namespaceSelector: matchLabels: name: "openshift-monitoring" - from: - namespaceSelector: matchLabels: network.openshift.io/policy-group: ingress podSelector: {} policyTypes: - Ingress Additional resources About network policy About the OpenShift SDN default CNI network provider About the OVN-Kubernetes default Container Network Interface (CNI) network provider
apiVersion: v1 kind: Namespace metadata: name: openshift-operators-redhat 1 annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-monitoring: "true" 2 oc apply -f <filename>.yaml apiVersion: v1 kind: Namespace metadata: name: openshift-logging annotations: openshift.io/node-selector: "" labels: openshift.io/cluster-monitoring: "true" oc apply -f <filename>.yaml apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: cluster-logging namespace: openshift-logging 1 spec: targetNamespaces: - openshift-logging 2 oc apply -f <filename>.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-logging namespace: openshift-logging 1 spec: channel: stable 2 name: cluster-logging source: redhat-operators 3 sourceNamespace: openshift-marketplace oc apply -f <filename>.yaml oc get csv -n <namespace> NAMESPACE NAME DISPLAY VERSION REPLACES PHASE openshift-logging clusterlogging.5.7.0-202007012112.p0 OpenShift Logging 5.7.0-202007012112.p0 Succeeded apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging spec: managementState: Managed 2 logStore: type: elasticsearch 3 retentionPolicy: 4 application: maxAge: 1d infra: maxAge: 7d audit: maxAge: 7d elasticsearch: nodeCount: 3 5 storage: storageClassName: <storage_class_name> 6 size: 200G resources: 7 limits: memory: 16Gi requests: memory: 16Gi proxy: 8 resources: limits: memory: 256Mi requests: memory: 256Mi redundancyPolicy: SingleRedundancy visualization: type: kibana 9 kibana: replicas: 1 collection: type: fluentd 10 fluentd: {} oc get deployment NAME READY UP-TO-DATE AVAILABLE AGE cluster-logging-operator 1/1 1 1 18h elasticsearch-cd-x6kdekli-1 1/1 1 1 6m54s elasticsearch-cdm-x6kdekli-1 1/1 1 1 18h elasticsearch-cdm-x6kdekli-2 1/1 1 1 6m49s elasticsearch-cdm-x6kdekli-3 1/1 1 1 6m44s oc get pods -n openshift-logging NAME READY STATUS RESTARTS AGE cluster-logging-operator-66f77ffccb-ppzbg 1/1 Running 0 7m elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp 2/2 Running 0 2m40s elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc 2/2 Running 0 2m36s elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2 2/2 Running 0 2m4s collector-587vb 1/1 Running 0 2m26s collector-7mpb9 1/1 Running 0 2m30s collector-flm6j 1/1 Running 0 2m33s collector-gn4rn 1/1 Running 0 2m26s collector-nlgb6 1/1 Running 0 2m30s collector-snpkt 1/1 Running 0 2m28s kibana-d6d5668c5-rppqm 2/2 Running 0 2m39s apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance 1 namespace: openshift-logging 2 spec: managementState: Managed 3 apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: spec: logStore: type: <log_store_type> 1 elasticsearch: 2 nodeCount: <integer> resources: {} storage: {} redundancyPolicy: <redundancy_type> 3 lokistack: 4 name: {} apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: name: instance namespace: openshift-logging spec: managementState: Managed logStore: type: lokistack lokistack: name: logging-loki oc apply -f <filename>.yaml apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: spec: collection: type: <log_collector_type> 1 resources: {} tolerations: {} oc apply -f <filename>.yaml apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: spec: visualization: type: <visualizer_type> 1 kibana: 2 resources: {} nodeSelector: {} proxy: {} replicas: {} tolerations: {} ocpConsole: 3 logsLimit: {} timeout: {} oc apply -f <filename>.yaml oc adm pod-network join-projects --to=openshift-operators-redhat openshift-logging oc label namespace openshift-operators-redhat project=openshift-operators-redhat apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-from-openshift-monitoring-ingress-operators-redhat spec: ingress: - from: - podSelector: {} - from: - namespaceSelector: matchLabels: project: "openshift-operators-redhat" - from: - namespaceSelector: matchLabels: name: "openshift-monitoring" - from: - namespaceSelector: matchLabels: network.openshift.io/policy-group: ingress podSelector: {} policyTypes: - Ingress