Chapter 2. Configuring IPv6 single or dual-stack networking
Chapter 2. Configuring IPv6 single or dual-stack networking You can use the IPv6 networking protocol in either single-stack or dual-stack networking modes. 2.1. IPv6 networking with MicroShift The MicroShift service defaults to IPv4 address families cluster-wide. However, IPv6 single-stack and IPv4/IPv6 dual-stack networking is available on supported platforms. When you set the values for IPv6 in the MicroShift configuration file and restart the service, settings managed by the OVN-Kubernetes network plugin are updated automatically. After migrating to dual-stack networking, both new and existing pods have dual-stack networking enabled. If you require cluster-wide IPv6 access, such as for the control plane and other services, use the following configuration examples. The MicroShift Multus Container Network Interface (CNI) plugin can enable IPv6 for pods. For dual-stack networking, each MicroShift cluster network and service network supports up to two values in the cluster and service network configuration parameters. Important Plan for IPv6 before starting MicroShift for the first time. Switching a cluster to and from different IP families is not supported unless you are migrating a cluster from default single-stack to dual-stack networking. If you configure your networking for either IPv6 single stack or IPv4/IPv6 dual stack, you must restart application pods and services. Otherwise pods and services remain configured with the default IP family. 2.2. Configuring IPv6 single-stack networking You can use the IPv6 network protocol by updating the MicroShift service configuration file. Prerequisites You installed the OpenShift CLI ( oc ). You have root access to the cluster. Your cluster uses the OVN-Kubernetes network plugin. The host has an IPv6 address and IPv6 routes, including the default. Procedure If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml . Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts. Note After you create it, the config.yaml file takes precedence over built-in settings. Replace the default values in the network section of the MicroShift YAML with your valid values. Example single-stack IPv6 networking configuration apiServer: # ... network: clusterNetwork: - fd01::/48 1 serviceNetwork: - fd02::/112 2 node: nodeIP: 2600:1f14:1c48:ee00:2d76:3190:5bc2:5aef 3 # ... 1 Specify a clusterNetwork with a CIDR value that is less than 64 . 2 Specify an IPv6 CIDR with a prefix of 112 . Kubernetes uses only the lowest 16 bits. For a prefix of 112 , IP addresses are assigned from 112 to 128 bits. 3 Example node IP address. Valid values are IP addresses in the IPv6 address family. You must only specify an IPv6 address when an IPv4 network is also present. If an IPv4 network is not present, the MicroShift service automatically fills in this value upon restart. Complete any other configurations you require, then start MicroShift by running the following command: USD sudo systemctl start microshift Verification Retrieve the networks defined in the node resource by running the following command: USD oc get node -o jsonpath='{.items[].spec.podCIDRs[]}' Example output fd01::/48 Retrieve the status of the pods by running the following command: USD oc get pod -A -o wide Example output NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system csi-snapshot-controller-bb7cb654b-rqrt6 1/1 Running 0 65s fd01:0:0:1::5 microshift-9 <none> <none> openshift-dns dns-default-cjn66 2/2 Running 0 62s fd01:0:0:1::9 microshift-9 <none> <none> openshift-dns node-resolver-ppnjb 1/1 Running 0 63s 2001:db9:ca7:ff::1db8 microshift-9 <none> <none> openshift-ingress router-default-6d97d7b8b6-wdtmg 1/1 Running 0 61s fd01:0:0:1::8 microshift-9 <none> <none> openshift-ovn-kubernetes ovnkube-master-gfvp5 4/4 Running 0 63s 2001:db9:ca7:ff::1db8 microshift-9 <none> <none> openshift-ovn-kubernetes ovnkube-node-bnpjh 1/1 Running 0 63s 2001:db9:ca7:ff::1db8 microshift-9 <none> <none> openshift-service-ca service-ca-5d7bd9db6-j25bd 1/1 Running 0 60s fd01:0:0:1::4 microshift-9 <none> <none> openshift-storage lvms-operator-656cd9b59b-bwr47 1/1 Running 0 63s fd01:0:0:1::7 microshift-9 <none> <none> openshift-storage vg-manager-f7dmk 1/1 Running 0 27s fd01:0:0:1::a microshift-9 <none> <none> Retrieve the status of services by running the following command: USD oc get svc -A Example output NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP fd02::1 <none> 443/TCP 3m42s openshift-dns dns-default ClusterIP fd02::a <none> 53/UDP,53/TCP,9154/TCP 2m58s openshift-ingress router-default LoadBalancer fd02::f2e6 2001:db9:ca7:ff::1db8,fd01:0:0:1::2,fd02::1:0,fd69::2 80:31133/TCP,443:31996/TCP 2m58s openshift-ingress router-internal-default ClusterIP fd02::c55e <none> 80/TCP,443/TCP,1936/TCP 2m58s openshift-storage lvms-operator-metrics-service ClusterIP fd02::7afb <none> 443/TCP 2m58s openshift-storage lvms-webhook-service ClusterIP fd02::d8dd <none> 443/TCP 2m58s openshift-storage vg-manager-metrics-service ClusterIP fd02::fc1 <none> 443/TCP 2m58s 2.3. Configuring IPv6 dual-stack networking before MicroShift starts You can configure your MicroShift cluster to run on dual-stack networking that supports IPv4 and IPv6 address families by using the configuration file before starting the service. The first IP family in the configuration is the primary IP stack in the cluster. After the cluster is running with dual-stack networking, enable application pods and add-on services for dual-stack by restarting them. Important The OVN-Kubernetes network plugin requires that both IPv4 and IPv6 default routes be on the same network device. IPv4 and IPv6 default routes on separate network devices is not supported. Important When using dual-stack networking where IPv6 is required, you cannot use IPv4-mapped IPv6 addresses, such as ::FFFF:198.51.100.1 . Prerequisites You installed the OpenShift CLI ( oc ). You have root access to the cluster. Your cluster uses the OVN-Kubernetes network plugin. The host has both IPv4 and IPv6 addresses and routes, including a default for each. The host has at least two L3 networks, IPv4 and IPv6. Procedure If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml . Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts. Note After you create it, the config.yaml file takes precedence over built-in settings. If you have not started MicroShift, replace the default values in the network section of the MicroShift YAML with your valid values. Example dual-stack IPv6 networking configuration with network assignments apiServer: # ... apiServer: subjectAltNames: - 192.168.113.117 - 2001:db9:ca7:ff::1db8 network: clusterNetwork: - 10.42.0.0/16 - fd01::/48 1 serviceNetwork: - 10.43.0.0/16 - fd02::/112 2 node: nodeIP: 192.168.113.117 3 nodeIPv6: 2001:db9:ca7:ff::1db8 4 # ... 1 Specify an IPv6 clusterNetwork with a CIDR value that is less than 64 . 2 Specify an IPv6 CIDR with a prefix of 112 . Kubernetes uses only the lowest 16 bits. For a prefix of 112 , IP addresses are assigned from 112 to 128 bits. 3 Example node IP address. Must be an IPv4 address family. 4 Example node IP address for dual-stack configuration. Must be an IPv6 address family. Configurable only with dual-stack networking. Complete any other MicroShift configurations you require, then start MicroShift by running the following command: USD sudo systemctl start microshift Reset the IP family policy for application pods and services as needed, then restart those application pods and services to enable dual-stack networking. See "Resetting the IP family policy for application pods and services" for a simple example. Verification You can verify that all of the system services and pods to have two IP addresses, one for each family, by using the following steps: Retrieve the networks defined in the node resource by running the following command: USD oc get pod -n openshift-ingress router-default-5b75594b4-w7w6s -o jsonpath='{.status.podIPs}' Example output [{"ip":"10.42.0.4"},{"ip":"fd01:0:0:1::4"}] Retrieve the networks defined by the host network pods by running the following command: USD oc get pod -n openshift-ovn-kubernetes ovnkube-master-2fm2k -o jsonpath='{.status.podIPs}' Example output [{"ip":"192.168.113.117"},{"ip":"2001:db9:ca7:ff::1db8"}] 2.4. Migrating a MicroShift cluster to IPv6 dual-stack networking You can convert a single-stack cluster to dual-stack cluster networking that supports IPv4 and IPv6 address families by setting two entries in the service and cluster network parameters in the MicroShift configuration file. The first IP family in the configuration is the primary IP stack in the cluster. MicroShift system pods and services are automatically updated upon MicroShift restart. After the cluster is migrated to dual-stack networking and has restarted, enable workload pods and services for dual-stack networking by restarting them. Important The OVN-Kubernetes network plugin requires that both IPv4 and IPv6 default routes be on the same network device. IPv4 and IPv6 default routes on separate network devices is not supported. Important When using dual-stack networking where IPv6 is required, you cannot use IPv4-mapped IPv6 addresses, such as ::FFFF:198.51.100.1 . Prerequisites You installed the OpenShift CLI ( oc ). You have root access to the cluster. Your cluster uses the OVN-Kubernetes network plugin. The host has both IPv4 and IPv6 addresses and routes, including a default for each. The host has at least two L3 networks, IPv4 and IPv6. Procedure If you have not done so, make a copy of the provided config.yaml.default file in the /etc/microshift/ directory, renaming it config.yaml . Keep the new MicroShift config.yaml in the /etc/microshift/ directory. Your config.yaml file is read every time the MicroShift service starts. Note After you create it, the config.yaml file takes precedence over built-in settings. Add IPv6 configurations to the network section of the MicroShift YAML with your valid values: Warning You must keep the same first entry across restarts and migrations. This is true for any migration: single-to-dual stack, or dual-to-single stack. A complete wipe of the etcd database is required if a change to the first entry is needed. This might result in application data loss and is not supported. Add an IPv6 configuration for a second network in the network section of the MicroShift YAML with your valid values. Add network assignments to the network section of the MicroShift config.yaml to enable dual stack with IPv6 as secondary network. Example dual-stack IPv6 configuration with network assignments # ... apiServer: subjectAltNames: - 192.168.113.117 - 2001:db9:ca7:ff::1db8 1 network: clusterNetwork: - 10.42.0.0/16 2 - fd01::/48 3 serviceNetwork: - 10.43.0.0/16 - fd02::/112 4 node: nodeIP: 192.168.113.117 5 nodeIPv6: 2001:db9:ca7:ff::1db8 6 # ... 1 The IPv6 node address. 2 IPv4 network. Specify a clusterNetwork with a CIDR value that is less than 24 . 3 IPv6 network. Specify a clusterNetwork with a CIDR value that is less than 64 . 4 Specify an IPv6 CIDR with a prefix of 112 . Kubernetes uses only the lowest 16 bits. For a prefix of 112 , IP addresses are assigned from 112 to 128 bits. 5 Example node IP address. Maintain the IPv4 IP address. 6 Example node IP address. Must be an IPv6 address family. Complete any other configurations you require, then restart MicroShift by running the following command: USD sudo systemctl restart microshift Reset the IP family policy for application pods and services as needed, then restart those application pods and services to enable dual-stack networking. See "Resetting the IP family policy for application pods and services" for a simple example. Verification You can verify that all of the system services and pods to have two IP addresses, one for each family, by using the following steps: Retrieve the status of the pods by running the following command: USD oc get pod -A -o wide Example output NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system csi-snapshot-controller-bb7cb654b-7s5ql 1/1 Running 0 46m 10.42.0.6 microshift-9 <none> <none> openshift-dns dns-default-zxkqn 2/2 Running 0 46m 10.42.0.5 microshift-9 <none> <none> openshift-dns node-resolver-r2h5z 1/1 Running 0 46m 192.168.113.117 microshift-9 <none> <none> openshift-ingress router-default-5b75594b4-228z7 1/1 Running 0 2m5s 10.42.0.3 microshift-9 <none> <none> openshift-ovn-kubernetes ovnkube-master-bltk7 4/4 Running 2 (2m32s ago) 2m36s 192.168.113.117 microshift-9 <none> <none> openshift-ovn-kubernetes ovnkube-node-9ghgs 1/1 Running 2 (2m32s ago) 46m 192.168.113.117 microshift-9 <none> <none> openshift-service-ca service-ca-5d7bd9db6-qgwgw 1/1 Running 0 46m 10.42.0.7 microshift-9 <none> <none> openshift-storage lvms-operator-656cd9b59b-8rpf4 1/1 Running 0 46m 10.42.0.8 microshift-9 <none> <none> openshift-storage vg-manager-wqmh4 1/1 Running 2 (2m39s ago) 46m 10.42.0.10 microshift-9 <none> <none> Retrieve the networks defined by the OVN-K network plugin by running the following command: USD oc get pod -n openshift-ovn-kubernetes ovnkube-master-bltk7 -o jsonpath='{.status.podIPs}' Example output [{"ip":"192.168.113.117"},{"ip":"2001:db9:ca7:ff::1db8"}] Retrieve the networks defined in the node resource by running the following command: USD oc get pod -n openshift-ingress router-default-5b75594b4-228z7 -o jsonpath='{.status.podIPs}' Example output [{"ip":"10.42.0.3"},{"ip":"fd01:0:0:1::3"}] Note To return to single-stack networking, you can remove the second entry to the networks and return to the single stack that was configured before migrating to dual-stack. 2.5. Resetting the IP family policy for application pods and services The default ipFamilyPolicy configuration value, PreferSingleStack , does not automatically update in all services after you update your MicroShift configuration to dual-stack networking. To enable dual-stack networking in services and application pods, you must update the ipFamilyPolicy value. Prerequisites You used the MicroShift config.yaml to define a dual-stack network with an IPv6 address family. Procedure Set the spec.ipFamilyPolicy field to a valid value for dual-stack networking in your service or pod by using the following example: Example dual-stack network configuration for a service kind: Service apiVersion: v1 metadata: name: microshift-new-service labels: app: microshift-application spec: type: NodePort ipFamilyPolicy: `PreferDualStack` 1 # ... 1 Required. Valid values for dual-stack networking are PreferDualStack and RequireDualStack . The value you set depends on the requirements of your application. PreferSingleStack is the default value for the ipFamilyPolicy field. Restart any application pods that do not have a hostNetwork defined. Pods that do have a hostNetwork defined do not need to be restarted to update the ipFamilyPolicy value. Note MicroShift system services and pods are automatically updated when the ipFamilyPolicy value is updated. 2.6. OVN-Kubernetes IPv6 and dual-stack limitations The OVN-Kubernetes network plugin has the following limitations: For a cluster configured for dual-stack networking, both IPv4 and IPv6 traffic must use the same network interface as the default gateway. If this requirement is not met, pods on the host in the ovnkube-node daemon set enter the CrashLoopBackOff state. If you display a pod with a command such as oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-node -o yaml , the status field contains more than one message about the default gateway, as shown in the following output: I1006 16:09:50.985852 60651 helper_linux.go:73] Found default gateway interface br-ex 192.168.127.1 I1006 16:09:50.985923 60651 helper_linux.go:73] Found default gateway interface ens4 fe80::5054:ff:febe:bcd4 F1006 16:09:50.985939 60651 ovnkube.go:130] multiple gateway interfaces detected: br-ex ens4 The only resolution is to reconfigure the host networking so that both IP families use the same network interface for the default gateway. For a cluster configured for dual-stack networking, both the IPv4 and IPv6 routing tables must contain the default gateway. If this requirement is not met, pods on the host in the ovnkube-node daemon set enter the CrashLoopBackOff state. If you display a pod with a command such as oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-node -o yaml , the status field contains more than one message about the default gateway, as shown in the following output: I0512 19:07:17.589083 108432 helper_linux.go:74] Found default gateway interface br-ex 192.168.123.1 F0512 19:07:17.589141 108432 ovnkube.go:133] failed to get default gateway interface The only resolution is to reconfigure the host networking so that both IP families contain the default gateway. 2.7. Additional resources Using NetworkManager to disable IPv6 for a specific connection (Red Hat Enterprise Linux documentation)
apiServer: network: clusterNetwork: - fd01::/48 1 serviceNetwork: - fd02::/112 2 node: nodeIP: 2600:1f14:1c48:ee00:2d76:3190:5bc2:5aef 3 sudo systemctl start microshift oc get node -o jsonpath='{.items[].spec.podCIDRs[]}' fd01::/48 oc get pod -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system csi-snapshot-controller-bb7cb654b-rqrt6 1/1 Running 0 65s fd01:0:0:1::5 microshift-9 <none> <none> openshift-dns dns-default-cjn66 2/2 Running 0 62s fd01:0:0:1::9 microshift-9 <none> <none> openshift-dns node-resolver-ppnjb 1/1 Running 0 63s 2001:db9:ca7:ff::1db8 microshift-9 <none> <none> openshift-ingress router-default-6d97d7b8b6-wdtmg 1/1 Running 0 61s fd01:0:0:1::8 microshift-9 <none> <none> openshift-ovn-kubernetes ovnkube-master-gfvp5 4/4 Running 0 63s 2001:db9:ca7:ff::1db8 microshift-9 <none> <none> openshift-ovn-kubernetes ovnkube-node-bnpjh 1/1 Running 0 63s 2001:db9:ca7:ff::1db8 microshift-9 <none> <none> openshift-service-ca service-ca-5d7bd9db6-j25bd 1/1 Running 0 60s fd01:0:0:1::4 microshift-9 <none> <none> openshift-storage lvms-operator-656cd9b59b-bwr47 1/1 Running 0 63s fd01:0:0:1::7 microshift-9 <none> <none> openshift-storage vg-manager-f7dmk 1/1 Running 0 27s fd01:0:0:1::a microshift-9 <none> <none> oc get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP fd02::1 <none> 443/TCP 3m42s openshift-dns dns-default ClusterIP fd02::a <none> 53/UDP,53/TCP,9154/TCP 2m58s openshift-ingress router-default LoadBalancer fd02::f2e6 2001:db9:ca7:ff::1db8,fd01:0:0:1::2,fd02::1:0,fd69::2 80:31133/TCP,443:31996/TCP 2m58s openshift-ingress router-internal-default ClusterIP fd02::c55e <none> 80/TCP,443/TCP,1936/TCP 2m58s openshift-storage lvms-operator-metrics-service ClusterIP fd02::7afb <none> 443/TCP 2m58s openshift-storage lvms-webhook-service ClusterIP fd02::d8dd <none> 443/TCP 2m58s openshift-storage vg-manager-metrics-service ClusterIP fd02::fc1 <none> 443/TCP 2m58s apiServer: apiServer: subjectAltNames: - 192.168.113.117 - 2001:db9:ca7:ff::1db8 network: clusterNetwork: - 10.42.0.0/16 - fd01::/48 1 serviceNetwork: - 10.43.0.0/16 - fd02::/112 2 node: nodeIP: 192.168.113.117 3 nodeIPv6: 2001:db9:ca7:ff::1db8 4 sudo systemctl start microshift oc get pod -n openshift-ingress router-default-5b75594b4-w7w6s -o jsonpath='{.status.podIPs}' [{"ip":"10.42.0.4"},{"ip":"fd01:0:0:1::4"}] oc get pod -n openshift-ovn-kubernetes ovnkube-master-2fm2k -o jsonpath='{.status.podIPs}' [{"ip":"192.168.113.117"},{"ip":"2001:db9:ca7:ff::1db8"}] apiServer: subjectAltNames: - 192.168.113.117 - 2001:db9:ca7:ff::1db8 1 network: clusterNetwork: - 10.42.0.0/16 2 - fd01::/48 3 serviceNetwork: - 10.43.0.0/16 - fd02::/112 4 node: nodeIP: 192.168.113.117 5 nodeIPv6: 2001:db9:ca7:ff::1db8 6 sudo systemctl restart microshift oc get pod -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system csi-snapshot-controller-bb7cb654b-7s5ql 1/1 Running 0 46m 10.42.0.6 microshift-9 <none> <none> openshift-dns dns-default-zxkqn 2/2 Running 0 46m 10.42.0.5 microshift-9 <none> <none> openshift-dns node-resolver-r2h5z 1/1 Running 0 46m 192.168.113.117 microshift-9 <none> <none> openshift-ingress router-default-5b75594b4-228z7 1/1 Running 0 2m5s 10.42.0.3 microshift-9 <none> <none> openshift-ovn-kubernetes ovnkube-master-bltk7 4/4 Running 2 (2m32s ago) 2m36s 192.168.113.117 microshift-9 <none> <none> openshift-ovn-kubernetes ovnkube-node-9ghgs 1/1 Running 2 (2m32s ago) 46m 192.168.113.117 microshift-9 <none> <none> openshift-service-ca service-ca-5d7bd9db6-qgwgw 1/1 Running 0 46m 10.42.0.7 microshift-9 <none> <none> openshift-storage lvms-operator-656cd9b59b-8rpf4 1/1 Running 0 46m 10.42.0.8 microshift-9 <none> <none> openshift-storage vg-manager-wqmh4 1/1 Running 2 (2m39s ago) 46m 10.42.0.10 microshift-9 <none> <none> oc get pod -n openshift-ovn-kubernetes ovnkube-master-bltk7 -o jsonpath='{.status.podIPs}' [{"ip":"192.168.113.117"},{"ip":"2001:db9:ca7:ff::1db8"}] oc get pod -n openshift-ingress router-default-5b75594b4-228z7 -o jsonpath='{.status.podIPs}' [{"ip":"10.42.0.3"},{"ip":"fd01:0:0:1::3"}] kind: Service apiVersion: v1 metadata: name: microshift-new-service labels: app: microshift-application spec: type: NodePort ipFamilyPolicy: `PreferDualStack` 1 I1006 16:09:50.985852 60651 helper_linux.go:73] Found default gateway interface br-ex 192.168.127.1 I1006 16:09:50.985923 60651 helper_linux.go:73] Found default gateway interface ens4 fe80::5054:ff:febe:bcd4 F1006 16:09:50.985939 60651 ovnkube.go:130] multiple gateway interfaces detected: br-ex ens4 I0512 19:07:17.589083 108432 helper_linux.go:74] Found default gateway interface br-ex 192.168.123.1 F0512 19:07:17.589141 108432 ovnkube.go:133] failed to get default gateway interface