Chapter 10. Image configuration resources for ROSA with HCP
Chapter 10. Image configuration resources for ROSA with HCP Use the following procedure to configure image registries. 10.1. Image controller configuration parameters for ROSA with HCP The image.config.openshift.io/cluster resource holds cluster-wide information about how to handle images. The resource exists, but it is read only and can only be changed through supported tools like ROSA CLI ( rosa ). The canonical and only valid name is cluster . It can be configured in Red Hat OpenShift Service on AWS hosted control planes through ROSA CLI ( rosa ) commands. Note Parameters such as DisableScheduledImport , MaxImagesBulkImportedPerRepository , MaxScheduledImportsPerMinute , ScheduledImageImportMinimumIntervalSeconds , InternalRegistryHostname are not configurable. Parameters for ROSA CLI Description registry-config-allowed-registries Registries for which image pull and push actions are allowed. To specify all subdomains, add the asterisk ( * ) wildcard character as a prefix to the domain name. For example, *.example.com . You can specify an individual repository within a registry. For example, reg1.io/myrepo/myapp:latest . All other registries are blocked. The format should be a comma-separated list of allowed registries. For example, allowed.io, allowed.io2 . registry-config-insecure-registries Registries which do not have a valid TLS certificate or only support HTTP connections. To specify all subdomains, add the asterisk ( * ) wildcard character as a prefix to the domain name. For example, *.example.com . You can specify an individual repository within a registry. For example, reg1.io/myrepo/myapp:latest . The format should be a comma-separated list of insecure registries. For example, insecure.io, insecure.io2 . registry-config-blocked-registries Registries for which image pull and push actions are denied. To specify all subdomains, add the asterisk ( * ) wildcard character as a prefix to the domain name. For example, *.example.com . You can specify an individual repository within a registry. For example, reg1.io/myrepo/myapp:latest . All other registries are allowed. The format should be a comma-separated list of blocked registries. For example, blocked.io, blocked.io2 . registry-config-allowed-registries-for-import Contains configuration that determines how the container runtime should treat individual registries when accessing images for builds and pods. For example, whether or not to allow insecure access. It does not contain configuration for the internal cluster registry. Limits the container image registries from which normal users can import images. The format should be a comma-separated list of domainName:insecure . domainName specifies a domain name for the registry. insecure indicates whether the registry is secure or insecure. registry-config-additional-trusted-ca A JSON file containing the registry hostname as the key, and the PEM-encoded certificate as the value, for each additional registry CA to trust. Warning When the allowedRegistries parameter is defined, all registries are blocked unless explicitly listed. To prevent pod failure, a list of Red Hat registries is automatically whitelisted, as they are required by payload images within your environment. The current list consists of image-registry.openshift-image-registry.svc:5000,quay.io,registry.redhat.io and it is also visible when running the rosa describe cluster command. 10.2. Configuring image registry settings for ROSA with HCP You can configure image registry settings at cluster creation. The cluster's nodes will use the required configuration after creation. Procedure Create ROSA with HCP clusters with image registry by running the following command: USD rosa create cluster -cluster-name=<cluster_name> --sts --mode=auto \ --hosted-cp --operator-roles-prefix <operator_role_prefix> \ --oidc-config-id <id_of_oidc_configuration> \ --subnet-ids=<public_subnet_id>,<private_subnet_id> \ --registry-config-insecure-registries <insecure_registries> \ --registry-config-allowed-registries <allowed_registries> \ --registry-config-allowed-registries-for-import <registry_name:insecure> \ --registry-config-additional-trusted-ca <additional_trusted_ca_file> Note When using the allowedRegistries , blockedRegistries , or insecureRegistries parameter, you can specify an individual repository within a registry. For example: reg1.io/myrepo/myapp:latest . Avoid insecure external registries to reduce possible security risks. Parameters allowedRegistries , blockedRegistries are mutually exclusive. Verification Run the rosa describe command to verify that your image registry is enabled by running the following command: USD rosa describe cluster --cluster=<cluster_name> Example output Name: rosa-hcp-test Domain Prefix: rosa-hcp-test Display Name: rosa-hcp-test ID: <cluster_hcp_id> External ID: <cluster_hcp_id> Control Plane: ROSA Service Hosted OpenShift Version: 4.Y.Z Channel Group: stable DNS: <dns> AWS Account: <aws_id> AWS Billing Account: <aws_id> API URL: <ocm_api> Console URL: Region: us-east-1 Availability: - Control Plane: MultiAZ - Data Plane: SingleAZ Nodes: - Compute (desired): 2 - Compute (current): 2 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: /23 - Subnets: <subnet_ids> EC2 Metadata Http Tokens: optional Role (STS) ARN: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Installer-Role Support Role ARN: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Support-Role Instance IAM Roles: - Worker: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Worker-Role Operator IAM Roles: - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-capa-controller-manager - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-control-plane-operator - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-kms-provider - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-image-registry-installer-cloud-cred - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-cluster-csi-drivers-ebs-cloud-credent - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-cloud-network-config-controller-cloud Managed Policies: Yes State: ready Private: No Delete Protection: Disabled Created: Oct 01 2030 09:48:52 UTC User Workload Monitoring: Enabled OIDC Endpoint URL: https://<endpoint> (Managed) Audit Log Forwarding: Disabled External Authentication: Disabled Etcd Encryption: Disabled Registry Configuration: - Allowed Registries: <allowed_registry> 1 2 - Insecure Registries: <insecure_registry> 3 - Allowed Registries for Import: 4 - Domain Name: <domain_name> 5 - Insecure: true 6 - Platform Allowlist: <platform_allowlist_id> 7 - Registries: <list_of_registries> 8 - Additional Trusted CA: 9 - <registry_name> : REDACTED 1 Allowed Registries : A comma-separated list of registries for which image pull and push actions are allowed. 2 Blocked Registries : A comma-separated list of registries for which image pull and push actions are blocked. Parameters allowedRegistries , blockedRegistries are mutually exclusive. 3 Insecure Registries : A comma-separated list of registries which do not have a valid TLS certificate or only support HTTP connections. 4 Allowed Registries for Import : Limits the container image registries from which normal users can import images. The format should be a comma-separated list of domainName:insecure . 5 domainName : Specifies a domain name for the registry. 6 insecure : Indicates whether the registry is secure or insecure. 7 Platform Allowlist : A reference to the id of the list of registries that needs to be whitelisted for the platform to work. 8 Registries : The list of registries that needs to be whitelisted for the platform to work. 9 Additional Trusted CA : A JSON file containing the registry hostname as the key, and the PEM-encoded certificate as the value, for each additional registry CA to trust. List your nodes to check the applied changes by running the following command: USD oc get nodes Example output NAME STATUS ROLES AGE VERSION ip-10-0-137-182.us-east-2.compute.internal Ready,SchedulingDisabled worker 65m v1.31.3 ip-10-0-188-96.us-east-2.compute.internal Ready worker 65m v1.31.3 ip-10-0-200-59.us-east-2.compute.internal Ready worker 63m v1.31.3 10.3. Editing image registry settings for ROSA with HCP You can change the image registry config with the rosa edit command. Warning When the allowedRegistries parameter is defined, all registries are blocked unless explicitly listed. To prevent pod failure, a list of Red Hat registries is automatically whitelisted, as they are required by payload images within your environment. The current list consists of image-registry.openshift-image-registry.svc:5000,quay.io,registry.redhat.io and it is also visible when running the rosa describe cluster command. Note You can change any registry-related parameter, which will trigger a rollout across all machine pools; all machine pool nodes will be recreated, following pod draining from each node. Procedure Update or edit the image registry for the cluster by running the following command: USD rosa edit cluster --registry-config-insecure-registries <insecure_registries> \ --registry-config-allowed-registries <allowed_registries> \ --registry-config-allowed-registries-for-import <registry_name:insecure> \ --registry-config-additional-trusted-ca <additional_trusted_ca_file> Example output ? Changing any registry related parameter will trigger a rollout across all machinepools (all machinepool nodes will be recreated, following pod draining from each node). Do you want to proceed? Yes I: Updated cluster '<cluster_name>' Verification Run the rosa describe command again, to see if the changes you made to your image registry updated by running the following command: USD rosa describe cluster --cluster=<cluster_name> Example output Name: rosa-hcp-test Domain Prefix: rosa-hcp-test Display Name: rosa-hcp-test ID: <cluster_hcp_id> External ID: <cluster_hcp_id> Control Plane: ROSA Service Hosted OpenShift Version: 4.Y.Z Channel Group: stable DNS: <dns> AWS Account: <aws_id> AWS Billing Account: <aws_id> API URL: <ocm_api> Console URL: Region: us-east-1 Availability: - Control Plane: MultiAZ - Data Plane: SingleAZ Nodes: - Compute (desired): 2 - Compute (current): 2 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: /23 - Subnets: <subnet_ids> EC2 Metadata Http Tokens: optional Role (STS) ARN: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Installer-Role Support Role ARN: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Support-Role Instance IAM Roles: - Worker: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Worker-Role Operator IAM Roles: - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-capa-controller-manager - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-control-plane-operator - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-kms-provider - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-image-registry-installer-cloud-cred - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-cluster-csi-drivers-ebs-cloud-credent - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-cloud-network-config-controller-cloud Managed Policies: Yes State: ready Private: No Delete Protection: Disabled Created: Oct 01 2030 09:48:52 UTC User Workload Monitoring: Enabled OIDC Endpoint URL: https://<endpoint> (Managed) Audit Log Forwarding: Disabled External Authentication: Disabled Etcd Encryption: Disabled Registry Configuration: - Allowed Registries: <allowed_registry> 1 2 - Insecure Registries: <insecure_registry> 3 - Allowed Registries for Import: 4 - Domain Name: <domain_name> 5 - Insecure: true 6 - Platform Allowlist: <platform_allowlist_id> 7 - Registries: <list_of_registries> 8 - Additional Trusted CA: 9 - <registry_name> : REDACTED 1 Allowed Registries : A comma-separated list of registries for which image pull and push actions are allowed. 2 Blocked Registries : A comma-separated list of registries for which image pull and push actions are blocked. Parameters allowedRegistries , blockedRegistries are mutually exclusive. 3 Insecure Registries : A comma-separated list of registries which do not have a valid TLS certificate or only support HTTP connections. 4 Allowed Registries for Import : Limits the container image registries from which normal users can import images. The format should be a comma-separated list of domainName:insecure . 5 domainName : Specifies a domain name for the registry. 6 insecure : Indicates whether the registry is secure or insecure. 7 Platform Allowlist : A reference to the id of the list of registries that needs to be whitelisted for the platform to work. 8 Registries : The list of registries that needs to be whitelisted for the platform to work. 9 Additional Trusted CA : A JSON file containing the registry hostname as the key, and the PEM-encoded certificate as the value, for each additional registry CA to trust. 10.3.1. Updating platform allowlist for ROSA with HCP A list of Red Hat registries is automatically allowed and it is visible when running rosa describe cluster. This list can be periodically updated to ensure platform can be operated correctly. Impacted clusters will receive a notification with the new allowlist ID. In such cases, the user must use this parameter to update from the expected ID to the newly expected ID. Update or edit the image registry for the cluster by running the following command: USD rosa edit cluster --registry-config-platform-allowlist <newID>
rosa create cluster -cluster-name=<cluster_name> --sts --mode=auto --hosted-cp --operator-roles-prefix <operator_role_prefix> --oidc-config-id <id_of_oidc_configuration> --subnet-ids=<public_subnet_id>,<private_subnet_id> --registry-config-insecure-registries <insecure_registries> --registry-config-allowed-registries <allowed_registries> --registry-config-allowed-registries-for-import <registry_name:insecure> --registry-config-additional-trusted-ca <additional_trusted_ca_file> rosa describe cluster --cluster=<cluster_name> Name: rosa-hcp-test Domain Prefix: rosa-hcp-test Display Name: rosa-hcp-test ID: <cluster_hcp_id> External ID: <cluster_hcp_id> Control Plane: ROSA Service Hosted OpenShift Version: 4.Y.Z Channel Group: stable DNS: <dns> AWS Account: <aws_id> AWS Billing Account: <aws_id> API URL: <ocm_api> Console URL: Region: us-east-1 Availability: - Control Plane: MultiAZ - Data Plane: SingleAZ Nodes: - Compute (desired): 2 - Compute (current): 2 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: /23 - Subnets: <subnet_ids> EC2 Metadata Http Tokens: optional Role (STS) ARN: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Installer-Role Support Role ARN: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Support-Role Instance IAM Roles: - Worker: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Worker-Role Operator IAM Roles: - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-capa-controller-manager - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-control-plane-operator - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-kms-provider - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-image-registry-installer-cloud-cred - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-cluster-csi-drivers-ebs-cloud-credent - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-cloud-network-config-controller-cloud Managed Policies: Yes State: ready Private: No Delete Protection: Disabled Created: Oct 01 2030 09:48:52 UTC User Workload Monitoring: Enabled OIDC Endpoint URL: https://<endpoint> (Managed) Audit Log Forwarding: Disabled External Authentication: Disabled Etcd Encryption: Disabled Registry Configuration: - Allowed Registries: <allowed_registry> 1 2 - Insecure Registries: <insecure_registry> 3 - Allowed Registries for Import: 4 - Domain Name: <domain_name> 5 - Insecure: true 6 - Platform Allowlist: <platform_allowlist_id> 7 - Registries: <list_of_registries> 8 - Additional Trusted CA: 9 - <registry_name> : REDACTED oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-137-182.us-east-2.compute.internal Ready,SchedulingDisabled worker 65m v1.31.3 ip-10-0-188-96.us-east-2.compute.internal Ready worker 65m v1.31.3 ip-10-0-200-59.us-east-2.compute.internal Ready worker 63m v1.31.3 rosa edit cluster --registry-config-insecure-registries <insecure_registries> --registry-config-allowed-registries <allowed_registries> --registry-config-allowed-registries-for-import <registry_name:insecure> --registry-config-additional-trusted-ca <additional_trusted_ca_file> ? Changing any registry related parameter will trigger a rollout across all machinepools (all machinepool nodes will be recreated, following pod draining from each node). Do you want to proceed? Yes I: Updated cluster '<cluster_name>' rosa describe cluster --cluster=<cluster_name> Name: rosa-hcp-test Domain Prefix: rosa-hcp-test Display Name: rosa-hcp-test ID: <cluster_hcp_id> External ID: <cluster_hcp_id> Control Plane: ROSA Service Hosted OpenShift Version: 4.Y.Z Channel Group: stable DNS: <dns> AWS Account: <aws_id> AWS Billing Account: <aws_id> API URL: <ocm_api> Console URL: Region: us-east-1 Availability: - Control Plane: MultiAZ - Data Plane: SingleAZ Nodes: - Compute (desired): 2 - Compute (current): 2 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: /23 - Subnets: <subnet_ids> EC2 Metadata Http Tokens: optional Role (STS) ARN: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Installer-Role Support Role ARN: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Support-Role Instance IAM Roles: - Worker: arn:aws:iam::<aws_id>:role/<account_roles_prefix>-HCP-ROSA-Worker-Role Operator IAM Roles: - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-capa-controller-manager - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-control-plane-operator - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-kube-system-kms-provider - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-image-registry-installer-cloud-cred - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-cluster-csi-drivers-ebs-cloud-credent - arn:aws:iam::<aws_id>:role/<operator_roles_prefix>-openshift-cloud-network-config-controller-cloud Managed Policies: Yes State: ready Private: No Delete Protection: Disabled Created: Oct 01 2030 09:48:52 UTC User Workload Monitoring: Enabled OIDC Endpoint URL: https://<endpoint> (Managed) Audit Log Forwarding: Disabled External Authentication: Disabled Etcd Encryption: Disabled Registry Configuration: - Allowed Registries: <allowed_registry> 1 2 - Insecure Registries: <insecure_registry> 3 - Allowed Registries for Import: 4 - Domain Name: <domain_name> 5 - Insecure: true 6 - Platform Allowlist: <platform_allowlist_id> 7 - Registries: <list_of_registries> 8 - Additional Trusted CA: 9 - <registry_name> : REDACTED rosa edit cluster --registry-config-platform-allowlist <newID>