Chapter 4. Virtual builds with Red Hat Quay on OpenShift Container Platform
Chapter 4. Virtual builds with Red Hat Quay on OpenShift Container Platform The procedures in this section explain how to create an environment for bare metal builds for Red Hat Quay on OpenShift Container Platform. Virtual builds can be run on virtualized machines with Red Hat Quay on OpenShift Container Platform. With this method, the build manager first creates the Job Object resource. Then, the Job Object creates a pod using the quay-builder-image . The quay-builder-image contains the quay-builder binary and the Podman service. The created pod runs as unprivileged . The quay-builder binary then builds the image while communicating status and retrieving build information from the build manager . 4.1. Virtual builds limitations The following limitations apply to the virtual builds feature: Running virtual builds with Red Hat Quay on OpenShift Container Platform in an unprivileged context might cause some commands that were working under the build strategy to fail. Attempts to change the build strategy could potentially cause performance issues and reliability with the build. Running virtual builds directly in a container does not have the same isolation as using virtual machines. Changing the build environment might also cause builds that were previously working to fail. 4.2. Configuring virtual builds for Red Hat Quay on OpenShift Container Platform The procedures in this section explain how to create an environment for virtual builds for Red Hat Quay on OpenShift Container Platform. Note If you are using Amazon Web Service (AWS) S3 storage, you must modify your storage bucket in the AWS console, prior to running builders. See "Modifying your AWS S3 storage bucket" in the following section for the required parameters. If you are using a Google Cloud Platform (GCP) object bucket, you must configure cross-origin resource sharing (CORS) to enable virtual builds . Prerequisites You have an OpenShift Container Platform cluster provisioned with the Red Hat Quay Operator running. You have set the tls component to unmanaged and uploaded custom SSL/TLS certificates to the Red Hat Quay Operator. For more information, see SSL and TLS for Red Hat Quay . You have configured the OpenShift Container Platform TLS component for builds. You are logged into OpenShift Container Platform as a cluster administrator. Procedure Create a new project where your virtual builders will be run, for example, virtual-builders , by running the following command: USD oc new-project virtual-builders Create a ServiceAccount in the project that will be used to run builds by entering the following command: USD oc create sa -n virtual-builders quay-builder Example output serviceaccount/quay-builder created Provide the created service account with editing permissions so that it can run a build : USD oc adm policy -n virtual-builders add-role-to-user edit system:serviceaccount:virtual-builders:quay-builder Example output clusterrole.rbac.authorization.k8s.io/edit added: "system:serviceaccount:virtual-builders:quay-builder" Grant the builder worker anyuid scc permissions by entering the following command. This requires cluster administrator privileges, which is required because builders must run as the Podman user for unprivileged or rootless builds to work. USD oc adm policy -n virtual-builders add-scc-to-user anyuid -z quay-builder Example output clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "quay-builder" Obtain the token for the builder service account by entering the following command: USD oc create token quay-builder -n virtual-builders Note When the token expires you will need to request a new token. Optionally, you can also add a custom expiration. For example, specify --duration 20160m to retain the token for two weeks. Example output eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ... Determine the builder route by entering the following command: USD oc get route -n quay-enterprise Example output NAME: example-registry-quay-builder HOST/PORT: example-registry-quay-builder-quay-enterprise.apps.stevsmit-cluster-new.gcp.quaydev.org PATH: SERVICES: example-registry-quay-app PORT: grpc TERMINATION: passthrough/Redirect WILDCARD: None Generate a self-signed SSL/TlS certificate with the .crt extension by entering the following command: USD oc extract cm/kube-root-ca.crt -n openshift-apiserver Example output ca.crt Rename the ca.crt file to build-cluster.crt by entering the following command: USD mv ca.crt build-cluster.crt Update the config.yaml file of your Red Hat Quay on OpenShift Container Platform deployment to include an appropriate virtual builds configuration by using the OpenShift Container Platform web console. Click Operators Installed Operators Red Hat Quay Quay Registry . Click the name of your registry, for example, example-registry . Under Config Bundle Secret , click the name of your configuration bundle, for example, extra-ca-certificate-config-bundle-secret . Click Actions Edit Secret . Add an appropriate virtual builds configuration using the following as a reference: FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - <superusername> FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_BUILD_SUPPORT: True BUILDMAN_HOSTNAME: <sample_build_route> 1 BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 3600 2 ORCHESTRATOR: REDIS_HOST: <sample_redis_hostname> 3 REDIS_PASSWORD: "" REDIS_SSL: false REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetesPodman NAME: openshift BUILDER_NAMESPACE: <sample_builder_namespace> 4 SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty} # Kubernetes resource options K8S_API_SERVER: <sample_k8s_api_server> 5 K8S_API_TLS_CA: <sample_crt_file> 6 VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 1G 7 CONTAINER_CPU_LIMITS: 300m 8 CONTAINER_MEMORY_REQUEST: 1G 9 CONTAINER_CPU_REQUEST: 300m 10 NODE_SELECTOR_LABEL_KEY: "" NODE_SELECTOR_LABEL_VALUE: "" SERVICE_ACCOUNT_NAME: <sample_service_account_name> SERVICE_ACCOUNT_TOKEN: <sample_account_token> 11 HTTP_PROXY: <http://10.0.0.1:80> HTTPS_PROXY: <http://10.0.0.1:80> NO_PROXY: <hostname.example.com> 1 The build route is obtained by running USD oc get route -n with the namespace of your Red Hat Quay on OpenShift Container Platform deployment. A port must be provided at the end of the route, and it should use the following format: [quayregistry-cr-name]-quay-builder-[ocp-namespace].[ocp-domain-name]:443 . 2 If the JOB_REGISTRATION_TIMEOUT parameter is set too low, you might receive the following error: failed to register job to build manager: rpc error: code = Unauthenticated desc = Invalid build token: Signature has expired . This parameter should be set to at least 240 . 3 If your Redis host has a password or SSL/TLS certificates, you must update this field accordingly. 4 Set to match the name of your virtual builds namespace. This example used virtual-builders . 5 The K8S_API_SERVER is obtained by running USD oc cluster-info . 6 You must manually create and add your custom CA cert, for example, K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt . 7 Defaults to 5120Mi if left unspecified. 8 For virtual builds , you must ensure that there are enough resources in your cluster. Defaults to 1000m if left unspecified. 9 Defaults to 3968Mi if left unspecified. 10 Defaults to 500m if left unspecified. 11 Obtained when running USD oc create sa . Example virtual builds configuration FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - quayadmin FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_BUILD_SUPPORT: True BUILDMAN_HOSTNAME: example-registry-quay-builder-quay-enterprise.apps.docs.quayteam.org:443 BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 3600 ORCHESTRATOR: REDIS_HOST: example-registry-quay-redis REDIS_PASSWORD: "" REDIS_SSL: false REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetesPodman NAME: openshift BUILDER_NAMESPACE: virtual-builders SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty} # Kubernetes resource options K8S_API_SERVER: api.docs.quayteam.org:6443 K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 1G CONTAINER_CPU_LIMITS: 300m CONTAINER_MEMORY_REQUEST: 1G CONTAINER_CPU_REQUEST: 300m NODE_SELECTOR_LABEL_KEY: "" NODE_SELECTOR_LABEL_VALUE: "" SERVICE_ACCOUNT_NAME: quay-builder SERVICE_ACCOUNT_TOKEN: "eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ" HTTP_PROXY: <http://10.0.0.1:80> HTTPS_PROXY: <http://10.0.0.1:80> NO_PROXY: <hostname.example.com> Click Save on the Edit Secret page. Restart your Red Hat Quay on OpenShift Container Platform registry with the new configuration. 4.2.1. Modifying your AWS S3 storage bucket If you are using AWS S3 storage, you must change your storage bucket in the AWS console prior to starting a build . Procedure Log in to your AWS console at s3.console.aws.com . In the search bar, search for S3 and then click S3 . Click the name of your bucket, for example, myawsbucket . Click the Permissions tab. Under Cross-origin resource sharing (CORS) , include the following parameters: [ { "AllowedHeaders": [ "Authorization" ], "AllowedMethods": [ "GET" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [], "MaxAgeSeconds": 3000 }, { "AllowedHeaders": [ "Content-Type", "x-amz-acl", "origin" ], "AllowedMethods": [ "PUT" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [], "MaxAgeSeconds": 3000 } ] 4.2.2. Modifying your Google Cloud Platform object bucket Note Currently, modifying your Google Cloud Platform object bucket is not supported on IBM Power and IBM Z. Use the following procedure to configure cross-origin resource sharing (CORS) for virtual builders. Without CORS configuration, uploading a build Dockerfile fails. Procedure Use the following reference to create a JSON file for your specific CORS needs. For example: USD cat gcp_cors.json Example output [ { "origin": ["*"], "method": ["GET"], "responseHeader": ["Authorization"], "maxAgeSeconds": 3600 }, { "origin": ["*"], "method": ["PUT"], "responseHeader": [ "Content-Type", "x-goog-acl", "origin"], "maxAgeSeconds": 3600 } ] Enter the following command to update your GCP storage bucket: USD gcloud storage buckets update gs://<bucket_name> --cors-file=./gcp_cors.json Example output Updating Completed 1 You can display the updated CORS configuration of your GCP bucket by running the following command: USD gcloud storage buckets describe gs://<bucket_name> --format="default(cors)" Example output cors: - maxAgeSeconds: 3600 method: - GET origin: - '*' responseHeader: - Authorization - maxAgeSeconds: 3600 method: - PUT origin: - '*' responseHeader: - Content-Type - x-goog-acl - origin
oc new-project virtual-builders oc create sa -n virtual-builders quay-builder serviceaccount/quay-builder created oc adm policy -n virtual-builders add-role-to-user edit system:serviceaccount:virtual-builders:quay-builder clusterrole.rbac.authorization.k8s.io/edit added: "system:serviceaccount:virtual-builders:quay-builder" oc adm policy -n virtual-builders add-scc-to-user anyuid -z quay-builder clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "quay-builder" oc create token quay-builder -n virtual-builders eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ oc get route -n quay-enterprise NAME: example-registry-quay-builder HOST/PORT: example-registry-quay-builder-quay-enterprise.apps.stevsmit-cluster-new.gcp.quaydev.org PATH: SERVICES: example-registry-quay-app PORT: grpc TERMINATION: passthrough/Redirect WILDCARD: None oc extract cm/kube-root-ca.crt -n openshift-apiserver ca.crt mv ca.crt build-cluster.crt FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - <superusername> FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_BUILD_SUPPORT: True BUILDMAN_HOSTNAME: <sample_build_route> 1 BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 3600 2 ORCHESTRATOR: REDIS_HOST: <sample_redis_hostname> 3 REDIS_PASSWORD: "" REDIS_SSL: false REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetesPodman NAME: openshift BUILDER_NAMESPACE: <sample_builder_namespace> 4 SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty} # Kubernetes resource options K8S_API_SERVER: <sample_k8s_api_server> 5 K8S_API_TLS_CA: <sample_crt_file> 6 VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 1G 7 CONTAINER_CPU_LIMITS: 300m 8 CONTAINER_MEMORY_REQUEST: 1G 9 CONTAINER_CPU_REQUEST: 300m 10 NODE_SELECTOR_LABEL_KEY: "" NODE_SELECTOR_LABEL_VALUE: "" SERVICE_ACCOUNT_NAME: <sample_service_account_name> SERVICE_ACCOUNT_TOKEN: <sample_account_token> 11 HTTP_PROXY: <http://10.0.0.1:80> HTTPS_PROXY: <http://10.0.0.1:80> NO_PROXY: <hostname.example.com> FEATURE_USER_INITIALIZE: true BROWSER_API_CALLS_XHR_ONLY: false SUPER_USERS: - quayadmin FEATURE_USER_CREATION: false FEATURE_QUOTA_MANAGEMENT: true FEATURE_BUILD_SUPPORT: True BUILDMAN_HOSTNAME: example-registry-quay-builder-quay-enterprise.apps.docs.quayteam.org:443 BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 3600 ORCHESTRATOR: REDIS_HOST: example-registry-quay-redis REDIS_PASSWORD: "" REDIS_SSL: false REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetesPodman NAME: openshift BUILDER_NAMESPACE: virtual-builders SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty} # Kubernetes resource options K8S_API_SERVER: api.docs.quayteam.org:6443 K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 1G CONTAINER_CPU_LIMITS: 300m CONTAINER_MEMORY_REQUEST: 1G CONTAINER_CPU_REQUEST: 300m NODE_SELECTOR_LABEL_KEY: "" NODE_SELECTOR_LABEL_VALUE: "" SERVICE_ACCOUNT_NAME: quay-builder SERVICE_ACCOUNT_TOKEN: "eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ" HTTP_PROXY: <http://10.0.0.1:80> HTTPS_PROXY: <http://10.0.0.1:80> NO_PROXY: <hostname.example.com> [ { "AllowedHeaders": [ "Authorization" ], "AllowedMethods": [ "GET" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [], "MaxAgeSeconds": 3000 }, { "AllowedHeaders": [ "Content-Type", "x-amz-acl", "origin" ], "AllowedMethods": [ "PUT" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [], "MaxAgeSeconds": 3000 } ] cat gcp_cors.json [ { "origin": ["*"], "method": ["GET"], "responseHeader": ["Authorization"], "maxAgeSeconds": 3600 }, { "origin": ["*"], "method": ["PUT"], "responseHeader": [ "Content-Type", "x-goog-acl", "origin"], "maxAgeSeconds": 3600 } ] gcloud storage buckets update gs://<bucket_name> --cors-file=./gcp_cors.json Updating Completed 1 gcloud storage buckets describe gs://<bucket_name> --format="default(cors)" cors: - maxAgeSeconds: 3600 method: - GET origin: - '*' responseHeader: - Authorization - maxAgeSeconds: 3600 method: - PUT origin: - '*' responseHeader: - Content-Type - x-goog-acl - origin