Chapter 8. Ingress Node Firewall Operator in OpenShift Container Platform
Chapter 8. Ingress Node Firewall Operator in OpenShift Container Platform The Ingress Node Firewall Operator provides a stateless, eBPF-based firewall for managing node-level ingress traffic in OpenShift Container Platform. 8.1. Ingress Node Firewall Operator The Ingress Node Firewall Operator provides ingress firewall rules at a node level by deploying the daemon set to nodes you specify and manage in the firewall configurations. To deploy the daemon set, you create an IngressNodeFirewallConfig custom resource (CR). The Operator applies the IngressNodeFirewallConfig CR to create ingress node firewall daemon set daemon , which run on all nodes that match the nodeSelector . You configure rules of the IngressNodeFirewall CR and apply them to clusters using the nodeSelector and setting values to "true". Important The Ingress Node Firewall Operator supports only stateless firewall rules. The maximum transmission units (MTU) parameter is 4Kb (kilobytes) in OpenShift Container Platform 4.13. Network interface controllers (NICs) that do not support native XDP drivers will run at a lower performance. Ingress Node Firewall Operator is not supported on Amazon Web Services (AWS) with the default OpenShift installation or on Red Hat OpenShift Service on AWS (ROSA). For more information on Red Hat OpenShift Service on AWS support and ingress, see Ingress Operator in Red Hat OpenShift Service on AWS . 8.2. Installing the Ingress Node Firewall Operator As a cluster administrator, you can install the Ingress Node Firewall Operator by using the OpenShift Container Platform CLI or the web console. 8.2.1. Installing the Ingress Node Firewall Operator using the CLI As a cluster administrator, you can install the Operator using the CLI. Prerequisites You have installed the OpenShift CLI ( oc ). You have an account with administrator privileges. Procedure To create the openshift-ingress-node-firewall namespace, enter the following command: USD cat << EOF| oc create -f - apiVersion: v1 kind: Namespace metadata: labels: pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/enforce-version: v1.24 name: openshift-ingress-node-firewall EOF To create an OperatorGroup CR, enter the following command: USD cat << EOF| oc create -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: ingress-node-firewall-operators namespace: openshift-ingress-node-firewall EOF Subscribe to the Ingress Node Firewall Operator. To create a Subscription CR for the Ingress Node Firewall Operator, enter the following command: USD cat << EOF| oc create -f - apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ingress-node-firewall-sub namespace: openshift-ingress-node-firewall spec: name: ingress-node-firewall channel: stable source: redhat-operators sourceNamespace: openshift-marketplace EOF To verify that the Operator is installed, enter the following command: USD oc get ip -n openshift-ingress-node-firewall Example output NAME CSV APPROVAL APPROVED install-5cvnz ingress-node-firewall.4.13.0-202211122336 Automatic true To verify the version of the Operator, enter the following command: USD oc get csv -n openshift-ingress-node-firewall Example output NAME DISPLAY VERSION REPLACES PHASE ingress-node-firewall.4.13.0-202211122336 Ingress Node Firewall Operator 4.13.0-202211122336 ingress-node-firewall.4.13.0-202211102047 Succeeded 8.2.2. Installing the Ingress Node Firewall Operator using the web console As a cluster administrator, you can install the Operator using the web console. Prerequisites You have installed the OpenShift CLI ( oc ). You have an account with administrator privileges. Procedure Install the Ingress Node Firewall Operator: In the OpenShift Container Platform web console, click Operators OperatorHub . Select Ingress Node Firewall Operator from the list of available Operators, and then click Install . On the Install Operator page, under Installed Namespace , select Operator recommended Namespace . Click Install . Verify that the Ingress Node Firewall Operator is installed successfully: Navigate to the Operators Installed Operators page. Ensure that Ingress Node Firewall Operator is listed in the openshift-ingress-node-firewall project with a Status of InstallSucceeded . Note During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message. If the Operator does not have a Status of InstallSucceeded , troubleshoot using the following steps: Inspect the Operator Subscriptions and Install Plans tabs for any failures or errors under Status . Navigate to the Workloads Pods page and check the logs for pods in the openshift-ingress-node-firewall project. Check the namespace of the YAML file. If the annotation is missing, you can add the annotation workload.openshift.io/allowed=management to the Operator namespace with the following command: USD oc annotate ns/openshift-ingress-node-firewall workload.openshift.io/allowed=management Note For single-node OpenShift clusters, the openshift-ingress-node-firewall namespace requires the workload.openshift.io/allowed=management annotation. 8.3. Deploying Ingress Node Firewall Operator Prerequisite The Ingress Node Firewall Operator is installed. Procedure To deploy the Ingress Node Firewall Operator, create a IngressNodeFirewallConfig custom resource that will deploy the Operator's daemon set. You can deploy one or multiple IngressNodeFirewall CRDs to nodes by applying firewall rules. Create the IngressNodeFirewallConfig inside the openshift-ingress-node-firewall namespace named ingressnodefirewallconfig . Run the following command to deploy Ingress Node Firewall Operator rules: USD oc apply -f rule.yaml 8.3.1. Ingress Node Firewall configuration object The fields for the Ingress Node Firewall configuration object are described in the following table: Table 8.1. Ingress Node Firewall Configuration object Field Type Description metadata.name string The name of the CR object. The name of the firewall rules object must be ingressnodefirewallconfig . metadata.namespace string Namespace for the Ingress Firewall Operator CR object. The IngressNodeFirewallConfig CR must be created inside the openshift-ingress-node-firewall namespace. spec.nodeSelector string A node selection constraint used to target nodes through specified node labels. For example: spec: nodeSelector: node-role.kubernetes.io/worker: "" Note One label used in nodeSelector must match a label on the nodes in order for the daemon set to start. For example, if the node labels node-role.kubernetes.io/worker and node-type.kubernetes.io/vm are applied to a node, then at least one label must be set using nodeSelector for the daemon set to start. Note The Operator consumes the CR and creates an ingress node firewall daemon set on all the nodes that match the nodeSelector . Ingress Node Firewall Operator example configuration A complete Ingress Node Firewall Configuration is specified in the following example: Example Ingress Node Firewall Configuration object apiVersion: ingressnodefirewall.openshift.io/v1alpha1 kind: IngressNodeFirewallConfig metadata: name: ingressnodefirewallconfig namespace: openshift-ingress-node-firewall spec: nodeSelector: node-role.kubernetes.io/worker: "" Note The Operator consumes the CR and creates an ingress node firewall daemon set on all the nodes that match the nodeSelector . 8.3.2. Ingress Node Firewall rules object The fields for the Ingress Node Firewall rules object are described in the following table: Table 8.2. Ingress Node Firewall rules object Field Type Description metadata.name string The name of the CR object. interfaces array The fields for this object specify the interfaces to apply the firewall rules to. For example, - en0 and - en1 . nodeSelector array You can use nodeSelector to select the nodes to apply the firewall rules to. Set the value of your named nodeselector labels to true to apply the rule. ingress object ingress allows you to configure the rules that allow outside access to the services on your cluster. Ingress object configuration The values for the ingress object are defined in the following table: Table 8.3. ingress object Field Type Description sourceCIDRs array Allows you to set the CIDR block. You can configure multiple CIDRs from different address families. Note Different CIDRs allow you to use the same order rule. In the case that there are multiple IngressNodeFirewall objects for the same nodes and interfaces with overlapping CIDRs, the order field will specify which rule is applied first. Rules are applied in ascending order. rules array Ingress firewall rules.order objects are ordered starting at 1 for each source.CIDR with up to 100 rules per CIDR. Lower order rules are executed first. rules.protocolConfig.protocol supports the following protocols: TCP, UDP, SCTP, ICMP and ICMPv6. ICMP and ICMPv6 rules can match against ICMP and ICMPv6 types or codes. TCP, UDP, and SCTP rules can match against a single destination port or a range of ports using <start : end-1> format. Set rules.action to allow to apply the rule or deny to disallow the rule. Note Ingress firewall rules are verified using a verification webhook that blocks any invalid configuration. The verification webhook prevents you from blocking any critical cluster services such as the API server or SSH. Ingress Node Firewall rules object example A complete Ingress Node Firewall configuration is specified in the following example: Example Ingress Node Firewall configuration apiVersion: ingressnodefirewall.openshift.io/v1alpha1 kind: IngressNodeFirewall metadata: name: ingressnodefirewall spec: interfaces: - eth0 nodeSelector: matchLabels: <ingress_firewall_label_name>: <label_value> 1 ingress: - sourceCIDRs: - 172.16.0.0/12 rules: - order: 10 protocolConfig: protocol: ICMP icmp: icmpType: 8 #ICMP Echo request action: Deny - order: 20 protocolConfig: protocol: TCP tcp: ports: "8000-9000" action: Deny - sourceCIDRs: - fc00:f853:ccd:e793::0/64 rules: - order: 10 protocolConfig: protocol: ICMPv6 icmpv6: icmpType: 128 #ICMPV6 Echo request action: Deny 1 A <label_name> and a <label_value> must exist on the node and must match the nodeselector label and value applied to the nodes you want the ingressfirewallconfig CR to run on. The <label_value> can be true or false . By using nodeSelector labels, you can target separate groups of nodes to apply different rules to using the ingressfirewallconfig CR. Zero trust Ingress Node Firewall rules object example Zero trust Ingress Node Firewall rules can provide additional security to multi-interface clusters. For example, you can use zero trust Ingress Node Firewall rules to drop all traffic on a specific interface except for SSH. A complete configuration of a zero trust Ingress Node Firewall rule set is specified in the following example: Important Users need to add all ports their application will use to their allowlist in the following case to ensure proper functionality. Example zero trust Ingress Node Firewall rules apiVersion: ingressnodefirewall.openshift.io/v1alpha1 kind: IngressNodeFirewall metadata: name: ingressnodefirewall-zero-trust spec: interfaces: - eth1 1 nodeSelector: matchLabels: <ingress_firewall_label_name>: <label_value> 2 ingress: - sourceCIDRs: - 0.0.0.0/0 3 rules: - order: 10 protocolConfig: protocol: TCP tcp: ports: 22 action: Allow - order: 20 action: Deny 4 1 Network-interface cluster 2 The <label_name> and <label_value> needs to match the nodeSelector label and value applied to the specific nodes with which you wish to apply the ingressfirewallconfig CR. 3 0.0.0.0/0 set to match any CIDR 4 action set to Deny 8.4. Viewing Ingress Node Firewall Operator rules Procedure Run the following command to view all current rules : USD oc get ingressnodefirewall Choose one of the returned <resource> names and run the following command to view the rules or configs: USD oc get <resource> <name> -o yaml 8.5. Troubleshooting the Ingress Node Firewall Operator Run the following command to list installed Ingress Node Firewall custom resource definitions (CRD): USD oc get crds | grep ingressnodefirewall Example output NAME READY UP-TO-DATE AVAILABLE AGE ingressnodefirewallconfigs.ingressnodefirewall.openshift.io 2022-08-25T10:03:01Z ingressnodefirewallnodestates.ingressnodefirewall.openshift.io 2022-08-25T10:03:00Z ingressnodefirewalls.ingressnodefirewall.openshift.io 2022-08-25T10:03:00Z Run the following command to view the state of the Ingress Node Firewall Operator: USD oc get pods -n openshift-ingress-node-firewall Example output NAME READY STATUS RESTARTS AGE ingress-node-firewall-controller-manager 2/2 Running 0 5d21h ingress-node-firewall-daemon-pqx56 3/3 Running 0 5d21h The following fields provide information about the status of the Operator: READY , STATUS , AGE , and RESTARTS . The STATUS field is Running when the Ingress Node Firewall Operator is deploying a daemon set to the assigned nodes. Run the following command to collect all ingress firewall node pods' logs: USD oc adm must-gather - gather_ingress_node_firewall The logs are available in the sos node's report containing eBPF bpftool outputs at /sos_commands/ebpf . These reports include lookup tables used or updated as the ingress firewall XDP handles packet processing, updates statistics, and emits events.
cat << EOF| oc create -f - apiVersion: v1 kind: Namespace metadata: labels: pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/enforce-version: v1.24 name: openshift-ingress-node-firewall EOF cat << EOF| oc create -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: ingress-node-firewall-operators namespace: openshift-ingress-node-firewall EOF cat << EOF| oc create -f - apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: ingress-node-firewall-sub namespace: openshift-ingress-node-firewall spec: name: ingress-node-firewall channel: stable source: redhat-operators sourceNamespace: openshift-marketplace EOF oc get ip -n openshift-ingress-node-firewall NAME CSV APPROVAL APPROVED install-5cvnz ingress-node-firewall.4.13.0-202211122336 Automatic true oc get csv -n openshift-ingress-node-firewall NAME DISPLAY VERSION REPLACES PHASE ingress-node-firewall.4.13.0-202211122336 Ingress Node Firewall Operator 4.13.0-202211122336 ingress-node-firewall.4.13.0-202211102047 Succeeded oc annotate ns/openshift-ingress-node-firewall workload.openshift.io/allowed=management oc apply -f rule.yaml spec: nodeSelector: node-role.kubernetes.io/worker: "" apiVersion: ingressnodefirewall.openshift.io/v1alpha1 kind: IngressNodeFirewallConfig metadata: name: ingressnodefirewallconfig namespace: openshift-ingress-node-firewall spec: nodeSelector: node-role.kubernetes.io/worker: "" apiVersion: ingressnodefirewall.openshift.io/v1alpha1 kind: IngressNodeFirewall metadata: name: ingressnodefirewall spec: interfaces: - eth0 nodeSelector: matchLabels: <ingress_firewall_label_name>: <label_value> 1 ingress: - sourceCIDRs: - 172.16.0.0/12 rules: - order: 10 protocolConfig: protocol: ICMP icmp: icmpType: 8 #ICMP Echo request action: Deny - order: 20 protocolConfig: protocol: TCP tcp: ports: "8000-9000" action: Deny - sourceCIDRs: - fc00:f853:ccd:e793::0/64 rules: - order: 10 protocolConfig: protocol: ICMPv6 icmpv6: icmpType: 128 #ICMPV6 Echo request action: Deny apiVersion: ingressnodefirewall.openshift.io/v1alpha1 kind: IngressNodeFirewall metadata: name: ingressnodefirewall-zero-trust spec: interfaces: - eth1 1 nodeSelector: matchLabels: <ingress_firewall_label_name>: <label_value> 2 ingress: - sourceCIDRs: - 0.0.0.0/0 3 rules: - order: 10 protocolConfig: protocol: TCP tcp: ports: 22 action: Allow - order: 20 action: Deny 4 oc get ingressnodefirewall oc get <resource> <name> -o yaml oc get crds | grep ingressnodefirewall NAME READY UP-TO-DATE AVAILABLE AGE ingressnodefirewallconfigs.ingressnodefirewall.openshift.io 2022-08-25T10:03:01Z ingressnodefirewallnodestates.ingressnodefirewall.openshift.io 2022-08-25T10:03:00Z ingressnodefirewalls.ingressnodefirewall.openshift.io 2022-08-25T10:03:00Z oc get pods -n openshift-ingress-node-firewall NAME READY STATUS RESTARTS AGE ingress-node-firewall-controller-manager 2/2 Running 0 5d21h ingress-node-firewall-daemon-pqx56 3/3 Running 0 5d21h oc adm must-gather - gather_ingress_node_firewall