Chapter 3. Adding additional Certificate Authorities for Red Hat Quay
Chapter 3. Adding additional Certificate Authorities for Red Hat Quay Certificate Authorities (CAs) are used by Red Hat Quay to verify SSL/TLS connections with external services, like ODIC providers, LDAP providers, storage providers, and so on. The following sections provide information about uploading additional CAs to Red Hat Quay depending on your deployment type. 3.1. Adding additional Certificate Authorities to the Red Hat Quay container The extra_ca_certs directory is the directory where additional Certificate Authorities (CAs) can be stored to extend the set of trusted certificates. These certificates are used by Red Hat Quay to verify SSL/TLS connections with external services. When deploying Red Hat Quay, you can place the necessary CAs in this directory to ensure that connections to services like LDAP, OIDC, and storage systems are properly secured and validated. For standalone Red Hat Quay deployments, you must create this directory and copy the additional CA certificates into that directory. Prerequisites You have a CA for the desired service. Procedure View the certificate to be added to the container by entering the following command: USD cat storage.crt Example output -----BEGIN CERTIFICATE----- MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV... -----END CERTIFICATE----- Create the extra_ca_certs in the /config folder of your Red Hat Quay directory by entering the following command: USD mkdir -p /path/to/quay_config_folder/extra_ca_certs Copy the CA file to the extra_ca_certs folder. For example: USD cp storage.crt /path/to/quay_config_folder/extra_ca_certs/ Ensure that the storage.crt file exists within the extra_ca_certs folder by entering the following command: USD tree /path/to/quay_config_folder/extra_ca_certs Example output /path/to/quay_config_folder/extra_ca_certs ├── storage.crt---- Obtain the CONTAINER ID of your Quay consider by entering the following command: USD podman ps Example output CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS 5a3e82c4a75f <registry>/<repo>/quay:{productminv} "/sbin/my_init" 24 hours ago Up 18 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 443/tcp grave_keller Restart the container by entering the following command USD podman restart 5a3e82c4a75f Confirm that the certificate was copied into the container namespace by running the following command: USD podman exec -it 5a3e82c4a75f cat /etc/ssl/certs/storage.pem Example output -----BEGIN CERTIFICATE----- MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV... -----END CERTIFICATE----- 3.2. Adding additional Certificate Authorities to Red Hat Quay on OpenShift Container Platform On Red Hat Quay on OpenShift Container Platform, the extra_ca_certs configuration field is is used to populate additional Certificate Authorities (CAs) into the CA directory, which then adds the CAs into the system trust bundle. These certificates are used by Red Hat Quay to verify SSL/TLS connections with external services like LDAP, OIDC, and storage systems. When deploying or redeploying Red Hat Quay on OpenShift Container Platform, you can add one, or multiple, CAs into the CA directory to ensure that external services are properly secured and validated. On Red Hat Quay on OpenShift Container Platform deployments, you must manually add the extra_ca_certs configuration field to your config.yaml file and re-upload the config.yaml to OpenShift Container Platform. The following procedures show you how to download your existing configuration file, add additional CAs to your Red Hat Quay on OpenShift Container Platform deployment, and then re-upload the configuration file. 3.2.1. Downloading the existing configuration The following procedure shows you how to download the existing configuration by locating the Config Bundle Secret . Procedure Describe the QuayRegistry resource by entering the following command: USD oc describe quayregistry -n <quay_namespace> # ... Config Bundle Secret: example-registry-config-bundle-v123x # ... Obtain the secret data by entering the following command: USD oc get secret -n <quay_namespace> <example-registry-config-bundle-v123x> -o jsonpath='{.data}' Example output { "config.yaml": "RkVBVFVSRV9VU0 ... MDAwMAo=" } Decode the data by entering the following command: USD echo 'RkVBVFVSRV9VU0 ... MDAwMAo=' | base64 --decode Example output FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - quayadmin FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_PROXY_CACHE: true FEATURE_BUILD_SUPPORT: true DEFAULT_SYSTEM_REJECT_QUOTA_BYTES: 102400000 Optional. You can export the data into a YAML file into the current directory by passing in the >> config.yaml flag. For example: USD echo 'RkVBVFVSRV9VU0 ... MDAwMAo=' | base64 --decode >> config.yaml 3.2.2. Adding additional Certificate Authorities to Red Hat Quay on OpenShift Container Platform The following example shows you how to add additional Certificate Authorities to your Red Hat Quay on OpenShift Container Platform deployment. Prerequisites You have base64 decoded the original config bundle into a config.yaml file. For more information, see Downloading the existing configuration . You have a Certificate Authority (CA) file or files. Procedure Create a new YAML file, for example, extra-ca-certificate-config-bundle-secret.yaml : USD touch extra-ca-certificate-config-bundle-secret.yaml Create the extra-ca-certificate-config-bundle-secret resource. Create the resource by entering the following command: USD oc -n <namespace> create secret generic extra-ca-certificate-config-bundle-secret \ --from-file=config.yaml=</path/to/config.yaml> \ 1 --from-file=extra_ca_cert_<name-of-certificate-one>=<path/to/certificate_one> \ 2 --from-file=extra_ca_cert_<name-of-certificate-two>=<path/to/certificate_two> \ 3 --from-file=extra_ca_cert_<name-of-certificate-three>=<path/to/certificate_three> \ 4 --dry-run=client -o yaml > extra-ca-certificate-config-bundle-secret.yaml 1 Where <config.yaml> is your base64 decoded config.yaml file. 2 The extra CA file to be added to into the system trust bundle. 3 Optional. A second CA file to be added into the system trust bundle. 4 Optional. A third CA file to be added into the system trust bundle. Optional. You can check the content of the extra-ca-certificate-config-bundle-secret.yaml file by entering the following command: USD cat extra-ca-certificate-config-bundle-secret.yaml Example output apiVersion: v1 data: config.yaml: QUxMT1dfUFVMTFNfV0lUSE9VVF9TVFJJQ1RfTE9HR0lORzogZmFsc2UKQVVUSEVOVElDQVRJT05fVFlQRTogRGF0YWJhc2UKREVGQVVMVF9UQUdfRVhQSVJBVElPTjogMncKUFJFRkVSU... extra_ca_cert_certificate-one: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQyVENDQXNHZ0F3SUJBZ0lVS2xOai90VUJBZHBkNURjYkdRQUo4anRuKzd3d0RRWUpLb1pJaHZjTkFRRUwKQlFBd2ZERUxNQWtHQ... extra_ca_cert_certificate-three: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ0ekNDQXN1Z0F3SUJBZ0lVQmJpTXNUeExjM0s4ODNWby9GTThsWXlOS2lFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2ZERUxNQWtHQ... extra_ca_cert_certificate-two: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ0ekNDQXN1Z0F3SUJBZ0lVVFVPTXZ2YVdFOFRYV3djYTNoWlBCTnV2QjYwd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2ZERUxNQWtHQ... kind: Secret metadata: creationTimestamp: null name: custom-ssl-config-bundle-secret namespace: <namespace> Create the configBundleSecret resource by entering the following command: USD oc create -n <namespace> -f extra-ca-certificate-config-bundle-secret.yaml Example output secret/extra-ca-certificate-config-bundle-secret created Update the QuayRegistry YAML file to reference the extra-ca-certificate-config-bundle-secret object by entering the following command: USD oc patch quayregistry <registry_name> -n <namespace> --type=merge -p '{"spec":{"configBundleSecret":"extra-ca-certificate-config-bundle-secret"}}' Example output quayregistry.quay.redhat.com/example-registry patched Ensure that your QuayRegistry YAML file has been updated to use the extra CA certificate configBundleSecret resource by entering the following command: USD oc get quayregistry <registry_name> -n <namespace> -o yaml Example output # ... configBundleSecret: extra-ca-certificate-config-bundle-secret # ... 3.3. Adding custom SSL/TLS certificates when Red Hat Quay is deployed on Kubernetes When deployed on Kubernetes, Red Hat Quay mounts in a secret as a volume to store config assets. Currently, this breaks the upload certificate function of the superuser panel. As a temporary workaround, base64 encoded certificates can be added to the secret after Red Hat Quay has been deployed. Use the following procedure to add custom SSL/TLS certificates when Red Hat Quay is deployed on Kubernetes. Prerequisites Red Hat Quay has been deployed. You have a custom ca.crt file. Procedure Base64 encode the contents of an SSL/TLS certificate by entering the following command: USD cat ca.crt | base64 -w 0 Example output ...c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= Enter the following kubectl command to edit the quay-enterprise-config-secret file: USD kubectl --namespace quay-enterprise edit secret/quay-enterprise-config-secret Add an entry for the certificate and paste the full base64 encoded stringer under the entry. For example: custom-cert.crt: c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= Use the kubectl delete command to remove all Red Hat Quay pods. For example: USD kubectl delete pod quay-operator.v3.7.1-6f9d859bd-p5ftc quayregistry-clair-postgres-7487f5bd86-xnxpr quayregistry-quay-app-upgrade-xq2v6 quayregistry-quay-database-859d5445ff-cqthr quayregistry-quay-redis-84f888776f-hhgms Afterwards, the Red Hat Quay deployment automatically schedules replace pods with the new certificate data.
cat storage.crt -----BEGIN CERTIFICATE----- MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV -----END CERTIFICATE----- mkdir -p /path/to/quay_config_folder/extra_ca_certs cp storage.crt /path/to/quay_config_folder/extra_ca_certs/ tree /path/to/quay_config_folder/extra_ca_certs /path/to/quay_config_folder/extra_ca_certs ├── storage.crt---- podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS 5a3e82c4a75f <registry>/<repo>/quay:{productminv} "/sbin/my_init" 24 hours ago Up 18 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 443/tcp grave_keller podman restart 5a3e82c4a75f podman exec -it 5a3e82c4a75f cat /etc/ssl/certs/storage.pem -----BEGIN CERTIFICATE----- MIIDTTCCAjWgAwIBAgIJAMVr9ngjJhzbMA0GCSqGSIb3DQEBCwUAMD0xCzAJBgNV -----END CERTIFICATE----- oc describe quayregistry -n <quay_namespace> Config Bundle Secret: example-registry-config-bundle-v123x oc get secret -n <quay_namespace> <example-registry-config-bundle-v123x> -o jsonpath='{.data}' { "config.yaml": "RkVBVFVSRV9VU0 ... MDAwMAo=" } echo 'RkVBVFVSRV9VU0 ... MDAwMAo=' | base64 --decode FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - quayadmin FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_PROXY_CACHE: true FEATURE_BUILD_SUPPORT: true DEFAULT_SYSTEM_REJECT_QUOTA_BYTES: 102400000 echo 'RkVBVFVSRV9VU0 ... MDAwMAo=' | base64 --decode >> config.yaml touch extra-ca-certificate-config-bundle-secret.yaml oc -n <namespace> create secret generic extra-ca-certificate-config-bundle-secret --from-file=config.yaml=</path/to/config.yaml> \ 1 --from-file=extra_ca_cert_<name-of-certificate-one>=<path/to/certificate_one> \ 2 --from-file=extra_ca_cert_<name-of-certificate-two>=<path/to/certificate_two> \ 3 --from-file=extra_ca_cert_<name-of-certificate-three>=<path/to/certificate_three> \ 4 --dry-run=client -o yaml > extra-ca-certificate-config-bundle-secret.yaml cat extra-ca-certificate-config-bundle-secret.yaml apiVersion: v1 data: config.yaml: QUxMT1dfUFVMTFNfV0lUSE9VVF9TVFJJQ1RfTE9HR0lORzogZmFsc2UKQVVUSEVOVElDQVRJT05fVFlQRTogRGF0YWJhc2UKREVGQVVMVF9UQUdfRVhQSVJBVElPTjogMncKUFJFRkVSU extra_ca_cert_certificate-one: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQyVENDQXNHZ0F3SUJBZ0lVS2xOai90VUJBZHBkNURjYkdRQUo4anRuKzd3d0RRWUpLb1pJaHZjTkFRRUwKQlFBd2ZERUxNQWtHQ extra_ca_cert_certificate-three: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ0ekNDQXN1Z0F3SUJBZ0lVQmJpTXNUeExjM0s4ODNWby9GTThsWXlOS2lFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2ZERUxNQWtHQ extra_ca_cert_certificate-two: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ0ekNDQXN1Z0F3SUJBZ0lVVFVPTXZ2YVdFOFRYV3djYTNoWlBCTnV2QjYwd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2ZERUxNQWtHQ kind: Secret metadata: creationTimestamp: null name: custom-ssl-config-bundle-secret namespace: <namespace> oc create -n <namespace> -f extra-ca-certificate-config-bundle-secret.yaml secret/extra-ca-certificate-config-bundle-secret created oc patch quayregistry <registry_name> -n <namespace> --type=merge -p '{"spec":{"configBundleSecret":"extra-ca-certificate-config-bundle-secret"}}' quayregistry.quay.redhat.com/example-registry patched oc get quayregistry <registry_name> -n <namespace> -o yaml configBundleSecret: extra-ca-certificate-config-bundle-secret cat ca.crt | base64 -w 0 ...c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= kubectl --namespace quay-enterprise edit secret/quay-enterprise-config-secret custom-cert.crt: c1psWGpqeGlPQmNEWkJPMjJ5d0pDemVnR2QNCnRsbW9JdEF4YnFSdVd3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= kubectl delete pod quay-operator.v3.7.1-6f9d859bd-p5ftc quayregistry-clair-postgres-7487f5bd86-xnxpr quayregistry-quay-app-upgrade-xq2v6 quayregistry-quay-database-859d5445ff-cqthr quayregistry-quay-redis-84f888776f-hhgms