Chapter 16. Migrating a standalone Red Hat Quay deployment to a Red Hat Quay Operator deployment
Chapter 16. Migrating a standalone Red Hat Quay deployment to a Red Hat Quay Operator deployment The following procedures allow you to back up a standalone Red Hat Quay deployment and migrate it to the Red Hat Quay Operator on OpenShift Container Platform. 16.1. Backing up a standalone deployment of Red Hat Quay Procedure Back up the config.yaml of your standalone Red Hat Quay deployment: USD mkdir /tmp/quay-backup USD cp /path/to/Quay/config/directory/config.yaml /tmp/quay-backup Create a backup of the database that your standalone Red Hat Quay deployment is using: USD pg_dump -h DB_HOST -p 5432 -d QUAY_DATABASE_NAME -U QUAY_DATABASE_USER -W -O > /tmp/quay-backup/quay-database-backup.sql Install the AWS CLI if you do not have it already. Create an ~/.aws/ directory: USD mkdir ~/.aws/ Obtain the access_key and secret_key from the config.yaml of your standalone deployment: USD grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/config.yaml Example output: DISTRIBUTED_STORAGE_CONFIG: minio-1: - RadosGWStorage - access_key: ########## bucket_name: quay hostname: 172.24.10.50 is_secure: false port: "9000" secret_key: ########## storage_path: /datastorage/registry Store the access_key and secret_key from the config.yaml file in your ~/.aws directory: USD touch ~/.aws/credentials Optional: Check that your access_key and secret_key are stored: USD cat > ~/.aws/credentials << EOF [default] aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG EOF Example output: aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG Note If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file , you can, you can configure these by running aws configure and manually inputting the credentials. In your quay-backup directory, create a bucket_backup directory: USD mkdir /tmp/quay-backup/bucket-backup Backup all blobs from the S3 storage: USD aws s3 sync --no-verify-ssl --endpoint-url https://PUBLIC_S3_ENDPOINT:PORT s3://QUAY_BUCKET/ /tmp/quay-backup/bucket-backup/ Note The PUBLIC_S3_ENDPOINT can be read from the Red Hat Quay config.yaml file under hostname in the DISTRIBUTED_STORAGE_CONFIG . If the endpoint is insecure, use http instead of https in the endpoint URL. Up to this point, you should have a complete backup of all Red Hat Quay data, blobs, the database, and the config.yaml file stored locally. In the following section, you will migrate the standalone deployment backup to Red Hat Quay on OpenShift Container Platform. 16.2. Using backed up standalone content to migrate to OpenShift Container Platform. Prerequisites Your standalone Red Hat Quay data, blobs, database, and config.yaml have been backed up. Red Hat Quay is deployed on OpenShift Container Platform using the Red Hat Quay Operator. A QuayRegistry with all components set to managed . Procedure The procedure in this documents uses the following namespace: quay-enterprise . Scale down the Red Hat Quay Operator: USD oc scale --replicas=0 deployment quay-operator.v3.6.2 -n openshift-operators Scale down the application and mirror deployments: USD oc scale --replicas=0 deployment QUAY_MAIN_APP_DEPLOYMENT QUAY_MIRROR_DEPLOYMENT Copy the database SQL backup to the Quay PostgreSQL database instance: USD oc cp /tmp/user/quay-backup/quay-database-backup.sql quay-enterprise/quayregistry-quay-database-54956cdd54-p7b2w:/var/lib/pgsql/data/userdata Obtain the database password from the Operator-created config.yaml file: USD oc get deployment quay-quay-app -o json | jq '.spec.template.spec.volumes[].projected.sources' | grep -i config-secret Example output: "name": "QUAY_CONFIG_SECRET_NAME" USD oc get secret quay-quay-config-secret-9t77hb84tb -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > /tmp/quay-backup/operator-quay-config-yaml-backup.yaml cat /tmp/quay-backup/operator-quay-config-yaml-backup.yaml | grep -i DB_URI Example output: Execute a shell inside of the database pod: # oc exec -it quay-postgresql-database-pod -- /bin/bash Enter psql: bash-4.4USD psql Drop the database: postgres=# DROP DATABASE "example-restore-registry-quay-database"; Example output: Create a new database and set the owner as the same name: postgres=# CREATE DATABASE "example-restore-registry-quay-database" OWNER "example-restore-registry-quay-database"; Example output: Connect to the database: postgres=# \c "example-restore-registry-quay-database"; Example output: You are now connected to database "example-restore-registry-quay-database" as user "postgres". Create a pg_trmg extension of your Quay database: example-restore-registry-quay-database=# create extension pg_trgm ; Example output: CREATE EXTENSION Exit the postgres CLI to re-enter bash-4.4: \q Set the password for your PostgreSQL deployment: bash-4.4USD psql -h localhost -d "QUAY_DATABASE_NAME" -U QUAY_DATABASE_OWNER -W < /var/lib/pgsql/data/userdata/quay-database-backup.sql Example output: Exit bash mode: bash-4.4USD exit Create a new configuration bundle for the Red Hat Quay Operator. USD touch config-bundle.yaml In your new config-bundle.yaml , include all of the information that the registry requires, such as LDAP configuration, keys, and other modifications that your old registry had. Run the following command to move the secret_key to your config-bundle.yaml : USD cat /tmp/quay-backup/config.yaml | grep SECRET_KEY > /tmp/quay-backup/config-bundle.yaml Note You must manually copy all the LDAP, OIDC and other information and add it to the /tmp/quay-backup/config-bundle.yaml file. Create a configuration bundle secret inside of your OpenShift cluster: USD oc create secret generic new-custom-config-bundle --from-file=config.yaml=/tmp/quay-backup/config-bundle.yaml Scale up the Quay pods: Scale up the mirror pods: Patch the QuayRegistry CRD so that it contains the reference to the new custom configuration bundle: Note If Red Hat Quay returns a 500 internal server error, you might have to update the location of your DISTRIBUTED_STORAGE_CONFIG to default . Create a new AWS credentials.yaml in your /.aws/ directory and include the access_key and secret_key from the Operator-created config.yaml file: USD touch credentials.yaml USD grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/operator-quay-config-yaml-backup.yaml USD cat > ~/.aws/credentials << EOF [default] aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG EOF Note If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file , you can configure these by running aws configure and manually inputting the credentials. Record the NooBaa's publicly available endpoint: USD oc get route s3 -n openshift-storage -o yaml -o jsonpath="{.spec.host}{'\n'}" Sync the backup data to the NooBaa backend storage: USD aws s3 sync --no-verify-ssl --endpoint-url https://NOOBAA_PUBLIC_S3_ROUTE /tmp/quay-backup/bucket-backup/* s3://QUAY_DATASTORE_BUCKET_NAME Scale the Operator back up to 1 pod: USD oc scale -replicas=1 deployment quay-operator.v3.6.4 -n openshift-operators The Operator uses the custom configuration bundle provided and reconciles all secrets and deployments. Your new Red Hat Quay deployment on OpenShift Container Platform should contain all of the information that the old deployment had. You should be able to pull all images.
mkdir /tmp/quay-backup cp /path/to/Quay/config/directory/config.yaml /tmp/quay-backup pg_dump -h DB_HOST -p 5432 -d QUAY_DATABASE_NAME -U QUAY_DATABASE_USER -W -O > /tmp/quay-backup/quay-database-backup.sql mkdir ~/.aws/ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/config.yaml DISTRIBUTED_STORAGE_CONFIG: minio-1: - RadosGWStorage - access_key: ########## bucket_name: quay hostname: 172.24.10.50 is_secure: false port: "9000" secret_key: ########## storage_path: /datastorage/registry touch ~/.aws/credentials cat > ~/.aws/credentials << EOF [default] aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG EOF aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG mkdir /tmp/quay-backup/bucket-backup aws s3 sync --no-verify-ssl --endpoint-url https://PUBLIC_S3_ENDPOINT:PORT s3://QUAY_BUCKET/ /tmp/quay-backup/bucket-backup/ oc scale --replicas=0 deployment quay-operator.v3.6.2 -n openshift-operators oc scale --replicas=0 deployment QUAY_MAIN_APP_DEPLOYMENT QUAY_MIRROR_DEPLOYMENT oc cp /tmp/user/quay-backup/quay-database-backup.sql quay-enterprise/quayregistry-quay-database-54956cdd54-p7b2w:/var/lib/pgsql/data/userdata oc get deployment quay-quay-app -o json | jq '.spec.template.spec.volumes[].projected.sources' | grep -i config-secret "name": "QUAY_CONFIG_SECRET_NAME" oc get secret quay-quay-config-secret-9t77hb84tb -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > /tmp/quay-backup/operator-quay-config-yaml-backup.yaml cat /tmp/quay-backup/operator-quay-config-yaml-backup.yaml | grep -i DB_URI postgresql://QUAY_DATABASE_OWNER:PASSWORD@DATABASE_HOST/QUAY_DATABASE_NAME oc exec -it quay-postgresql-database-pod -- /bin/bash bash-4.4USD psql postgres=# DROP DATABASE "example-restore-registry-quay-database"; DROP DATABASE postgres=# CREATE DATABASE "example-restore-registry-quay-database" OWNER "example-restore-registry-quay-database"; CREATE DATABASE postgres=# \c "example-restore-registry-quay-database"; You are now connected to database "example-restore-registry-quay-database" as user "postgres". example-restore-registry-quay-database=# create extension pg_trgm ; CREATE EXTENSION \q bash-4.4USD psql -h localhost -d "QUAY_DATABASE_NAME" -U QUAY_DATABASE_OWNER -W < /var/lib/pgsql/data/userdata/quay-database-backup.sql SET SET SET SET SET bash-4.4USD exit touch config-bundle.yaml cat /tmp/quay-backup/config.yaml | grep SECRET_KEY > /tmp/quay-backup/config-bundle.yaml oc create secret generic new-custom-config-bundle --from-file=config.yaml=/tmp/quay-backup/config-bundle.yaml oc scale --replicas=1 deployment quayregistry-quay-app deployment.apps/quayregistry-quay-app scaled oc scale --replicas=1 deployment quayregistry-quay-mirror deployment.apps/quayregistry-quay-mirror scaled oc patch quayregistry QUAY_REGISTRY_NAME --type=merge -p '{"spec":{"configBundleSecret":"new-custom-config-bundle"}}' touch credentials.yaml grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/operator-quay-config-yaml-backup.yaml cat > ~/.aws/credentials << EOF [default] aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG EOF oc get route s3 -n openshift-storage -o yaml -o jsonpath="{.spec.host}{'\n'}" aws s3 sync --no-verify-ssl --endpoint-url https://NOOBAA_PUBLIC_S3_ROUTE /tmp/quay-backup/bucket-backup/* s3://QUAY_DATASTORE_BUCKET_NAME oc scale -replicas=1 deployment quay-operator.v3.6.4 -n openshift-operators