Chapter 8. Backup and recovery
Chapter 8. Backup and recovery Backing up CodeReady Workspaces involves a combination of the following processes that back up different data: Use the CodeReady Workspaces Operator and a crwctl . Note Use the internal backup server to test this process. Use backups of persistent volumes to back up and restore the source code stored in users' workspaces. Tip Whether or not you implement backups of persistent volumes, advise users to commit and push their changes to avoid losing their work. Use backups of the external PostgreSQL database to back up and restore persisting data about the state of CodeReady Workspaces. 8.1. Supported Restic-compatible backup servers CodeReady Workspaces uses the CodeReady Workspaces Operator and integrated Restic to back up and restore CodeReady Workspaces instances from backup snapshots on a configured backup server. The CodeReady Workspaces Operator automates the creation of a Restic backup repository on the backup server. To back up data, the CodeReady Workspaces Operator gathers the data required for a backup snapshot and uses Restic to create and manage the snapshot. To restore data, the CodeReady Workspaces Operator uses Restic to retrieve and decrypt the snapshot, and then the CodeReady Workspaces Operator applies the retrieved data to perform the recovery. CodeReady Workspaces can use the following backup servers that are compatible with the integrated Restic: SFTP See the documentation for the SFTP server solution you plan to use ( OpenSSH or a derived commercial product) and the Restic Docs on SFTP . Amazon S3 See the documentation for Amazon S3 (or the chosen S3 API compatible storage) and the Restic Docs on Amazon S3 . REST See the README for Rest Server and the Restic Docs on Rest Server . For testing the backing up and restoring of CodeReady Workspaces instances, CodeReady Workspaces offers the internal backup server . Additional resources Restic Documentation restic · Backups done right! OpenSSH.com How to configure an sftp server with restricted chroot users with ssh keys 8.2. Backing up of CodeReady Workspaces instances to an SFTP backup server You can send backups of CodeReady Workspaces instances to an SFTP backup server with custom resources or crwctl: Section 8.2.1, "Backing up a CodeReady Workspaces instance to an SFTP backup server by using custom resources" Section 8.2.2, "Backing up a CodeReady Workspaces instance to an SFTP backup server by using crwctl" 8.2.1. Backing up a CodeReady Workspaces instance to an SFTP backup server by using custom resources Backing up a CodeReady Workspaces instance to an SFTP backup server by using custom resources requires two custom objects: First you create a custom object to configure CodeReady Workspaces to use an SFTP backup server. Then you create a custom object to make and send a backup snapshot of a CodeReady Workspaces instance to the configured SFTP backup server. 8.2.1.1. Configuring CodeReady Workspaces with custom resources to use an SFTP backup server To configure CodeReady Workspaces to use an SFTP backup server: Prerequisites Configured SFTP backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Create a Secret containing the repo-password key with a password: apiVersion: v1 kind: Secret metadata: name: backup-encryption-password-secret namespace: eclipse-che type: Opaque stringData: repo-password: <password> (The CodeReady Workspaces Operator will set up this password for the backup repository that the CodeReady Workspaces Operator will create from this custom object on the backup server.) Warning The backup repository password is used to encrypt the backup data. If you lose this password, you will lose the backup data. Create a Secret (for example, name: ssh-key-secret ) containing the ssh-privatekey key with a private SSH key for logging in to the SFTP server without a password. Create the CheBackupServerConfiguration custom object: apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration namespace: openshift-workspaces spec: 1 sftp: username: <username> 2 hostname: <hostname> 3 port: 1234 4 repositoryPath: red-hat-codeready-workspaces-backups 5 repositoryPasswordSecretRef: backup-encryption-password-secret 6 sshKeySecretRef: ssh-key-secret 7 1 Must only contain one section (such as sftp ). 2 User name on the remote server to log in using the SSH protocol. 3 Remote server hostname. 4 Optional property that specifies the port on which an SFTP server is running. The default value is 22 . 5 Absolute or relative path on the server where backup snapshots are stored. 6 Secret created in step 1. 7 Secret created in step 2. Optional: To configure multiple backup servers, create a separate CheBackupServerConfiguration custom object for each backup server. Important The CodeReady Workspaces Operator automatically backs up the CodeReady Workspaces instance before every CodeReady Workspaces update, permitting rollback to CodeReady Workspaces version if needed. If you configure only one backup server, that backup server is automatically used for pre-update backups by default. If you configure multiple backup servers, you must add the che.eclipse.org/backup-before-update: true annotation to the custom object of only one of them to specify it as the default backup server for pre-update backups. (If you don't add this annotation for one of multiple backup servers, or if you add this annotation for multiple backup servers, then the CodeReady Workspaces Operator defaults to using the internal backup server for pre-update backups.) 8.2.1.2. Backing up a CodeReady Workspaces instance to an SFTP backup server by using the CheClusterBackup custom object You can use a CheClusterBackup custom object to make a backup snapshot of a CodeReady Workspaces instance and to send the snapshot to the configured backup server. To create each and every backup snapshot requires a new CheClusterBackup custom object; that is, editing an already consumed CheClusterBackup custom object, during or after backing up, has no effect. Warning This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes . Prerequisites Configured backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Created CheBackupServerConfiguration custom object. See the section of this guide. Procedure Create the CheClusterBackup custom object, which creates a backup snapshot: apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: red-hat-codeready-workspaces-backup namespace: openshift-workspaces spec: backupServerConfigRef: backup-server-configuration 1 useInternalBackupServer: false 2 1 Name of the CheBackupServerConfiguration custom object defining what backup server to use. 2 Configures the Operator through this custom resource to use the CodeReady Workspaces-managed internal backup server or an administrator-managed external backup server (SFTP, Amazon S3 or S3 API compatible storage, or REST). Tip If you intend to reuse a name for CheClusterBackup custom objects, first delete any existing custom object with the same name . To delete it on the command line, use oc: Read the status section of the CheClusterBackup custom object to verify the backup process, for example: status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 1 Summary of the latest state of the backup process. 2 Current stage of the backup process in a human-readable format. 3 Backup process state: InProgress or Succeeded or Failed . The CodeReady Workspaces instance is backed up in a snapshot when state is Succeeded : status: ... message: >- Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC m=+999.553272281 1 snapshotId: 9f0adce2 2 state: Succeeded 3 1 Summary of the latest state of the backup process. 2 ID of the created backup snapshot. This field appears only when state is Succeeded . 3 Backup process state: Succeeded (or Failed ). 8.2.2. Backing up a CodeReady Workspaces instance to an SFTP backup server by using crwctl You can use crwctl to make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to a configured SFTP backup server. To do so, enter crwctl with the command-line options or set the environment variables. 8.2.2.1. Backing up a CodeReady Workspaces instance to an SFTP backup server by using crwctl with command-line options To make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to a configured SFTP backup server, run crwctl with the command-line options. Warning This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes . Prerequisites Installed crwctl . Configured SFTP backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Enter the crwctl server:backup command with the following arguments: 1 Backup repository URL as an argument using the --repository-url or -r option. Syntax for the backup repository URL: sftp: <user> @ <host> :/ <repository_path_on_sftp_server> . An example of a repository path on the SFTP server: /srv/restic-repo . For more details about repository URL syntax, see Restic Documentation . 2 Backup repository password as an argument using the --repository-password or -p option. 3 Path to a private SSH key file for authenticating on the SFTP server. Tip Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore . Verify the output of the entered command. For example: 8.2.2.2. Backing up a CodeReady Workspaces instance to an SFTP backup server by using crwctl and a CheBackupServerConfiguration custom object You can back up a CodeReady Workspaces instance by entering the name of a CheBackupServerConfiguration custom object as an argument with the crwctl server:backup command. Prerequisites Installed crwctl . Configured backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Created CheBackupServerConfiguration custom object. Procedure Run the following command on a command line: 1 This option points crwctl to a CheBackupServerConfiguration custom object. You can find the name of the CheBackupServerConfiguration custom object under metadata in the custom object. Tip Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --backup-server-config-name option with the crwctl server:backup command just once and omit this option onward when entering crwctl server:backup . Verify the output of the entered command. For example: 8.2.2.3. Configuring crwctl for an SFTP backup server with environment variables As an alternative to using command-line options for crwctl server:backup , you can configure crwctl for an SFTP backup server with environment variables. This enables you to run crwctl server:backup without having to enter those options with it. Prerequisites Installed crwctl . Configured SFTP backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Which environment variables you need to set depends on your use case: If you do not have a CheBackupServerConfiguration custom object yet, then you can set the following environment variables: BACKUP_REPOSITORY_URL for the backup repository. BACKUP_REPOSITORY_PASSWORD for the backup repository password. One of the two environment variables for the SSH key (for logging in without a password): SSH_KEY_FILE for the path to the SSH key file. SSH_KEY for the SSH key. If you already have a CheBackupServerConfiguration custom object, you can set the following environment variable that will point crwctl to that CheBackupServerConfiguration custom object: BACKUP_SERVER_CONFIG_NAME for the name of a CheBackupServerConfiguration custom object to point crwctl to. (You can find name under metadata in the custom object.) 8.3. Backing up of CodeReady Workspaces instances to Amazon S3 You can send backups of CodeReady Workspaces instances to Amazon S3 (or S3 API compatible storage) with custom resources or crwctl: Section 8.3.1, "Backing up a CodeReady Workspaces instance to Amazon S3 by using custom resources" Section 8.3.2, "Backing up a CodeReady Workspaces instance to Amazon S3 by using crwctl" 8.3.1. Backing up a CodeReady Workspaces instance to Amazon S3 by using custom resources Backing up a CodeReady Workspaces instance to Amazon S3 (or S3 API compatible storage) by using custom resources requires two custom objects: First you create a custom object to configure CodeReady Workspaces to use Amazon S3. Then you create a custom object to make and send a backup snapshot of a CodeReady Workspaces instance to the configured Amazon S3. 8.3.1.1. Configuring CodeReady Workspaces with custom resources to use Amazon S3 To configure CodeReady Workspaces to use Amazon S3: Prerequisites Configured Amazon S3. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Create a Secret containing the repo-password key with a password: apiVersion: v1 kind: Secret metadata: name: backup-encryption-password-secret namespace: eclipse-che type: Opaque stringData: repo-password: <password> (The CodeReady Workspaces Operator will set up this password for the backup repository that the CodeReady Workspaces Operator will create from this custom object on the backup server.) Warning The backup repository password is used to encrypt the backup data. If you lose this password, you will lose the backup data. Create a Secret (for example, name: aws-user-credentials-secret ) containing: awsAccessKeyId key with the AWS access key ID of the user awsSecretAccessKey key with the AWS secret access key of the user Create the CheBackupServerConfiguration custom object: apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration namespace: openshift-workspaces spec: 1 awss3: protocol: https 2 hostname: s3.amazonaws.com 3 port: 443 4 repositoryPath: red-hat-codeready-workspaces-backups 5 repositoryPasswordSecretRef: backup-encryption-password-secret 6 awsAccessKeySecretRef: aws-user-credentials-secret 7 1 Must only contain one section (such as awss3 ). 2 Optional property that specifies the protocol to be used. https is the default value; http is a permitted value. 3 Optional property that specifies the S3 hostname. The default value is s3.amazonaws.com . 4 Optional property that specifies the port on which the backup server is running. The default value is 443 . 5 Name of the bucket resource that stores the backup snapshots. You must manually create the bucket resource before configuring it here. 6 Secret created in step 1. 7 Secret created in step 2. Optional: To configure multiple backup servers, create a separate CheBackupServerConfiguration custom object for each backup server. Important The CodeReady Workspaces Operator automatically backs up the CodeReady Workspaces instance before every CodeReady Workspaces update, permitting rollback to CodeReady Workspaces version if needed. If you configure only one backup server, that backup server is automatically used for pre-update backups by default. If you configure multiple backup servers, you must add the che.eclipse.org/backup-before-update: true annotation to the custom object of only one of them to specify it as the default backup server for pre-update backups. (If you don't add this annotation for one of multiple backup servers, or if you add this annotation for multiple backup servers, then the CodeReady Workspaces Operator defaults to using the internal backup server for pre-update backups.) 8.3.1.2. Backing up a CodeReady Workspaces instance to Amazon S3 by using the CheClusterBackup custom object You can use a CheClusterBackup custom object to make a backup snapshot of a CodeReady Workspaces instance and to send the snapshot to the configured backup server. To create each and every backup snapshot requires a new CheClusterBackup custom object; that is, editing an already consumed CheClusterBackup custom object, during or after backing up, has no effect. Warning This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes . Prerequisites Configured backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Created CheBackupServerConfiguration custom object. See the section of this guide. Procedure Create the CheClusterBackup custom object, which creates a backup snapshot: apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: red-hat-codeready-workspaces-backup namespace: openshift-workspaces spec: backupServerConfigRef: backup-server-configuration 1 useInternalBackupServer: false 2 1 Name of the CheBackupServerConfiguration custom object defining what backup server to use. 2 Configures the Operator through this custom resource to use the CodeReady Workspaces-managed internal backup server or an administrator-managed external backup server (SFTP, Amazon S3 or S3 API compatible storage, or REST). Tip If you intend to reuse a name for CheClusterBackup custom objects, first delete any existing custom object with the same name . To delete it on the command line, use oc: Read the status section of the CheClusterBackup custom object to verify the backup process, for example: status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 1 Summary of the latest state of the backup process. 2 Current stage of the backup process in a human-readable format. 3 Backup process state: InProgress or Succeeded or Failed . The CodeReady Workspaces instance is backed up in a snapshot when state is Succeeded : status: ... message: >- Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC m=+999.553272281 1 snapshotId: 9f0adce2 2 state: Succeeded 3 1 Summary of the latest state of the backup process. 2 ID of the created backup snapshot. This field appears only when state is Succeeded . 3 Backup process state: Succeeded (or Failed ). 8.3.2. Backing up a CodeReady Workspaces instance to Amazon S3 by using crwctl You can use crwctl to make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to the configured Amazon S3 (or S3 API compatible storage). To do so, enter crwctl with the command-line options or set the environment variables. 8.3.2.1. Backing up a CodeReady Workspaces instance to Amazon S3 by using crwctl with command-line options To make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to the configured Amazon S3, run crwctl with the command-line options. Warning This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes . Prerequisites Installed crwctl . Configured Amazon S3. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Enter the crwctl server:backup command with the following arguments: 1 Backup repository URL as an argument using the --repository-url or -r option. Syntax for the backup repository URL: s3:s3.amazonaws.com/ <bucket_name> . For more details about repository URL syntax, see Restic Documentation . 2 Backup repository password as an argument using the --repository-password or -p option. 3 User's AWS access key ID. 4 User's AWS secret access key. Tip Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore . Verify the output of the entered command. For example: 8.3.2.2. Backing up a CodeReady Workspaces instance to Amazon S3 by using crwctl and a CheBackupServerConfiguration custom object You can back up a CodeReady Workspaces instance by entering the name of a CheBackupServerConfiguration custom object as an argument with the crwctl server:backup command. Prerequisites Installed crwctl . Configured backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Created CheBackupServerConfiguration custom object. Procedure Run the following command on a command line: 1 This option points crwctl to a CheBackupServerConfiguration custom object. You can find the name of the CheBackupServerConfiguration custom object under metadata in the custom object. Tip Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --backup-server-config-name option with the crwctl server:backup command just once and omit this option onward when entering crwctl server:backup . Verify the output of the entered command. For example: 8.3.2.3. Configuring crwctl with environment variables to use Amazon S3 As an alternative to using command-line options for crwctl server:backup , you can configure crwctl for Amazon S3 with environment variables. This enables you to run crwctl server:backup without having to enter those options with it. Prerequisites Installed crwctl . Configured Amazon S3. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Which environment variables you need to set depends on your use case: If you do not have a CheBackupServerConfiguration custom object yet, then you can set the following environment variables: BACKUP_REPOSITORY_URL for the backup repository. BACKUP_REPOSITORY_PASSWORD for the backup repository password. AWS_ACCESS_KEY_ID for the user's AWS access key ID. AWS_SECRET_ACCESS_KEY for the user's AWS secret access key. If you already have a CheBackupServerConfiguration custom object, you can set the following environment variable that will point crwctl to that CheBackupServerConfiguration custom object: BACKUP_SERVER_CONFIG_NAME for the name of a CheBackupServerConfiguration custom object to point crwctl to. (You can find name under metadata in the custom object.) 8.4. Backing up of CodeReady Workspaces instances to a REST backup server You can send backups of CodeReady Workspaces instances to a REST backup server with custom resources or crwctl: Section 8.4.1, "Backing up a CodeReady Workspaces instance to a REST backup server by using custom resources" Section 8.4.2, "Backing up a CodeReady Workspaces instance to a REST backup server by using crwctl" 8.4.1. Backing up a CodeReady Workspaces instance to a REST backup server by using custom resources Backing up a CodeReady Workspaces instance to a REST backup server by using custom resources requires two custom objects: First you create a custom object to configure CodeReady Workspaces to use a REST backup server. Then you create a custom object to make and send a backup snapshot of a CodeReady Workspaces instance to the configured REST backup server. 8.4.1.1. Configuring CodeReady Workspaces with custom resources to use a REST backup server To configure CodeReady Workspaces to use a REST backup server: Prerequisites Configured REST backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Create a Secret containing the repo-password key with a password: apiVersion: v1 kind: Secret metadata: name: backup-encryption-password-secret namespace: eclipse-che type: Opaque stringData: repo-password: <password> (The CodeReady Workspaces Operator will set up this password for the backup repository that the CodeReady Workspaces Operator will create from this custom object on the backup server.) Warning The backup repository password is used to encrypt the backup data. If you lose this password, you will lose the backup data. Optional: Create a Secret (for example, name: rest-server-auth-secret ) containing the username and password keys for the REST server user credentials. Note Skip this step if not using the REST server user credentials. Create the CheBackupServerConfiguration custom object: apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration namespace: openshift-workspaces spec: 1 rest: protocol: https 2 hostname: <hostname> 3 port: 1234 4 repositoryPath: red-hat-codeready-workspaces-backups 5 repositoryPasswordSecretRef: backup-encryption-password-secret 6 credentialsSecretRef: rest-server-auth-secret 7 1 Must only contain one section (such as rest ). 2 Optional property that specifies the protocol to be used: https is the default value; http is a permitted value. 3 Backup server hostname. 4 Optional property that specifies the port on which the backup server is running. The default value is 8000 . 5 Path on the backup server where the backup snapshots are stored. 6 Secret created in step 1. 7 Optional property that specifies the Secret created in step 2. Optional: To configure multiple backup servers, create a separate CheBackupServerConfiguration custom object for each backup server. Important The CodeReady Workspaces Operator automatically backs up the CodeReady Workspaces instance before every CodeReady Workspaces update, permitting rollback to CodeReady Workspaces version if needed. If you configure only one backup server, that backup server is automatically used for pre-update backups by default. If you configure multiple backup servers, you must add the che.eclipse.org/backup-before-update: true annotation to the custom object of only one of them to specify it as the default backup server for pre-update backups. (If you don't add this annotation for one of multiple backup servers, or if you add this annotation for multiple backup servers, then the CodeReady Workspaces Operator defaults to using the internal backup server for pre-update backups.) 8.4.1.2. Backing up a CodeReady Workspaces instance to a REST backup server by using the CheClusterBackup custom object You can use a CheClusterBackup custom object to make a backup snapshot of a CodeReady Workspaces instance and to send the snapshot to the configured backup server. To create each and every backup snapshot requires a new CheClusterBackup custom object; that is, editing an already consumed CheClusterBackup custom object, during or after backing up, has no effect. Warning This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes . Prerequisites Configured backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Created CheBackupServerConfiguration custom object. See the section of this guide. Procedure Create the CheClusterBackup custom object, which creates a backup snapshot: apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: red-hat-codeready-workspaces-backup namespace: openshift-workspaces spec: backupServerConfigRef: backup-server-configuration 1 useInternalBackupServer: false 2 1 Name of the CheBackupServerConfiguration custom object defining what backup server to use. 2 Configures the Operator through this custom resource to use the CodeReady Workspaces-managed internal backup server or an administrator-managed external backup server (SFTP, Amazon S3 or S3 API compatible storage, or REST). Tip If you intend to reuse a name for CheClusterBackup custom objects, first delete any existing custom object with the same name . To delete it on the command line, use oc: Read the status section of the CheClusterBackup custom object to verify the backup process, for example: status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 1 Summary of the latest state of the backup process. 2 Current stage of the backup process in a human-readable format. 3 Backup process state: InProgress or Succeeded or Failed . The CodeReady Workspaces instance is backed up in a snapshot when state is Succeeded : status: ... message: >- Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC m=+999.553272281 1 snapshotId: 9f0adce2 2 state: Succeeded 3 1 Summary of the latest state of the backup process. 2 ID of the created backup snapshot. This field appears only when state is Succeeded . 3 Backup process state: Succeeded (or Failed ). 8.4.2. Backing up a CodeReady Workspaces instance to a REST backup server by using crwctl You can use crwctl to make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to a configured REST backup server. To do so, enter crwctl with the command-line options or set the environment variables. 8.4.2.1. Backing up a CodeReady Workspaces instance to a REST backup server by using crwctl with command-line options To make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to a configured REST backup server, run crwctl with the command-line options. Warning This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes . Prerequisites Installed crwctl . Configured REST backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Enter the crwctl server:backup command with the following arguments: 1 Backup repository URL as an argument using the --repository-url or -r option. Syntax for the backup repository URL: rest:http:// <host> :8000/ or rest:https:// <user> : <password> @ <host> :8000/ or rest:https:// <user> : <password> @ <host> :8000/ <repository_path_on_rest_server> / . For more details about repository URL syntax, see Restic Documentation . 2 Backup repository password as an argument using the --repository-password or -p option. 3 When authentication is required, the username for authenticating to the backup REST server as the --username= <username> argument. 4 When authentication is required, the password for authenticating to the backup REST server as the --password= <password> argument. Tip Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore . Verify the output of the entered command. For example: 8.4.2.2. Backing up a CodeReady Workspaces instance to a REST backup server by using crwctl and a CheBackupServerConfiguration custom object You can back up a CodeReady Workspaces instance by entering the name of a CheBackupServerConfiguration custom object as an argument with the crwctl server:backup command. Prerequisites Installed crwctl . Configured backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Created CheBackupServerConfiguration custom object. Procedure Run the following command on a command line: 1 This option points crwctl to a CheBackupServerConfiguration custom object. You can find the name of the CheBackupServerConfiguration custom object under metadata in the custom object. Tip Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --backup-server-config-name option with the crwctl server:backup command just once and omit this option onward when entering crwctl server:backup . Verify the output of the entered command. For example: 8.4.2.3. Configuring crwctl with environment variables to use a REST backup server As an alternative to using command-line options for crwctl server:backup , you can configure crwctl for a REST backup server with environment variables. This enables you to run crwctl server:backup without having to enter those options with it. Prerequisites Installed crwctl . Configured REST backup server. See Section 8.1, "Supported Restic-compatible backup servers" . Procedure Which environment variables you need to set depends on your use case: If you do not have a CheBackupServerConfiguration custom object yet, then you can set the following environment variables: BACKUP_REPOSITORY_URL for the backup repository. BACKUP_REPOSITORY_PASSWORD for the backup repository password. When authentication to the REST backup server is required: REST_SERVER_USERNAME for the username. REST_SERVER_PASSWORD for the password. If you already have a CheBackupServerConfiguration custom object, you can set the following environment variable that will point crwctl to that CheBackupServerConfiguration custom object: BACKUP_SERVER_CONFIG_NAME for the name of a CheBackupServerConfiguration custom object to point crwctl to. (You can find name under metadata in the custom object.) 8.5. Backing up of CodeReady Workspaces instances to the internal backup server To create backups for testing purposes, you can use the internal backup server. The internal backup server is a REST backup server that is managed by the CodeReady Workspaces Operator. Important The CodeReady Workspaces Operator defaults to using the internal backup server for pre-update backups (to allow the CodeReady Workspaces version rollback) if multiple backup servers are configured and if the che.eclipse.org/backup-before-update: true annotation is added to more than one of the multiple backup servers or is not added to the custom object of any one of the multiple backup servers. 8.5.1. Backing up a CodeReady Workspaces instance to the internal server by using the CheClusterBackup custom object You can create a custom object to back up a CodeReady Workspaces instance to the internal backup server. Note In this case, the CodeReady Workspaces Operator automatically configures the internal backup server and creates a CheBackupServerConfiguration custom object with name: backup-rest-server-configuration . Warning Configuring a backup server inside the same cluster, for example the internal backup server, is not the best practice because a cluster failure can be one of the scenarios when a backed up CodeReady Workspaces instance needs to be restored. Procedure Create the CheClusterBackup custom object to send a backup snapshot to the internal backup server: apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: red-hat-codeready-workspaces-backup namespace: openshift-workspaces spec: useInternalBackupServer: true 1 1 Configures the custom resource to back up to the internal backup server. Read the status section of the CheClusterBackup object to verify the backup process: ... status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 ... 1 Displays the overall state or error message. 2 Current phase of the backup process in a human-readable format. 3 Backup process state: InProgress or Succeeded or Failed . ... status: cheVersion: 7.37.0 1 message: >- 2 Backup successfully finished at 2021-10-07 11:18:28.116103737 +0000 UTC m=+2384.430366711 snapshotId: 40a09756 3 state: Succeeded 4 ... 1 The version of CodeReady Workspaces from which the backup snapshot was created. This field appears only when state is Succeeded . 2 Displays the overall state or error message. 3 The ID of the created backup snapshot. This field appears only when state is Succeeded . 4 Backup process state: InProgress or Succeeded or Failed . 8.5.2. Backing up a CodeReady Workspaces instance to the internal server by using crwctl You can use crwctl to take a backup snapshot of a CodeReady Workspaces instance and send it to the internal backup server. Note In this case, the CodeReady Workspaces Operator automatically configures the internal backup server and creates a CheBackupServerConfiguration custom object with name: backup-rest-server-configuration . Warning This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes . Prerequisites Installed crwctl . CodeReady Workspaces has not been configured to use a backup server. Procedure On a command line, enter the crwctl server:backup command with no arguments. Verify the output of the entered command. For example: 8.6. Restoring a CodeReady Workspaces instance from a backup You can restore a CodeReady Workspaces instance from a backup snapshot by using the CheClusterRestore custom object or crwctl : Section 8.6.1, "Restoring a CodeReady Workspaces instance from a backup by using the CheClusterRestore custom object" Section 8.6.2, "Restoring a CodeReady Workspaces instance from a backup by using crwctl" 8.6.1. Restoring a CodeReady Workspaces instance from a backup by using the CheClusterRestore custom object You can restore a CodeReady Workspaces instance from a backup snapshot by using custom resources. A new CheClusterRestore custom object is required every time you restore a CodeReady Workspaces instance from a backup snapshot. (So editing an already consumed CheClusterRestore custom object, during or after backing up, has no effect.) Warning Backup snapshots are bound to a OpenShift cluster. To restore a CodeReady Workspaces instance, you must only use a backup snapshot that was created in the same OpenShift cluster. Important Do not use the CheClusterRestore custom object to recover a CodeReady Workspaces instance of an earlier version of CodeReady Workspaces! Use only crwctl to recover a CodeReady Workspaces instance of an earlier version of CodeReady Workspaces; see Section 8.6.2, "Restoring a CodeReady Workspaces instance from a backup by using crwctl" ! Prerequisites Backup snapshot of a CodeReady Workspaces instance on any of the following: SFTP backup server Amazon S3 or S3 API compatible backup server REST backup server Internal backup server Procedure Create the CheClusterRestore custom object, which restores a backup: apiVersion: org.eclipse.che/v1 kind: CheClusterRestore metadata: name: red-hat-codeready-workspaces-restore namespace: openshift-workspaces spec: backupServerConfigRef: backup-server-configuration 1 snapshotId: <snapshot_id> 2 1 Name of the CheBackupServerConfiguration object that defines what backup server to use. You can find this name as backupServerConfigRef in spec of the CheClusterBackup custom object. (You can also find it as name in metadata of the CheBackupServerConfiguration custom object used to take the backup snapshot you are restoring.) 2 Optional parameter defining the Snapshot ID to restore from. The default value is the last snapshot in the backup repository on the backup server. Tip To reuse a name for CheClusterRestore custom objects, first delete any existing custom object with the same name . To delete it on the command line, use oc: USD oc delete CheClusterRestore <name> -n openshift-workspaces Read the status section of the CheClusterRestore object to monitor the recovery process, for example: status: message: 'Restore is in progress. Start time: <timestamp>' 1 stage: Restoring CodeReady Workspaces related cluster objects 2 state: InProgress 3 1 Overall state or error message; for example, Restore successfully finished . 2 Current phase of the recovery process in a human-readable format. 3 Recovery process state. One of InProgress , Succeeded , or Failed . Check that the CodeReady Workspaces instance has been recovered. Tip If errors occur in your browser after the recovery, clean up the browser data for the CodeReady Workspaces domain. Delete the CheClusterRestore object. 8.6.2. Restoring a CodeReady Workspaces instance from a backup by using crwctl Using crwctl , you can restore a CodeReady Workspaces instance from a backup snapshot by using any of the following options (as applicable): Section 8.6.2.1, "Restoring a CodeReady Workspaces instance from its latest backup by using crwctl" Section 8.6.2.2, "Restoring a CodeReady Workspaces instance from a backup by snapshot ID" Section 8.6.2.3, "Restoring a CodeReady Workspaces instance by using crwctl and a CheClusterBackup custom object" 8.6.2.1. Restoring a CodeReady Workspaces instance from its latest backup by using crwctl To restore the CodeReady Workspaces instance from its latest backup by using crwctl , you can enter latest as the snapshot ID. Warning Backup snapshots are bound to a OpenShift cluster. To restore a CodeReady Workspaces instance, you must only use a backup snapshot that was created in the same OpenShift cluster. Important If the backup snapshot was taken in an earlier version of Che, add the --version= <earlier_version_number> option to the crwctl server:restore command to roll back Che to that earlier version. Prerequisites Installed crwctl . Backup snapshot of a CodeReady Workspaces instance on any of the following: SFTP backup server Amazon S3 or S3 API compatible backup server REST backup server Internal backup server Procedure Run the crwctl server:restore command with the following arguments: 1 The latest snapshot. 2 The backup repository URL as an argument using the --repository-url or -r option. (Omit if using the internal backup server!) 3 The backup repository password as an argument using the --repository-password or -p option. (Omit if using the internal backup server!) Tip Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore . When prompted in the output, enter y to confirm: Tip To skip this question, add the --batch option to the server:restore command. Verify the output of the entered command. For example: 8.6.2.2. Restoring a CodeReady Workspaces instance from a backup by snapshot ID To restore a CodeReady Workspaces instance from a particular backup snapshot, you can enter the snapshot ID with crwctl . Warning Backup snapshots are bound to a OpenShift cluster. To restore a CodeReady Workspaces instance, you must only use a backup snapshot that was created in the same OpenShift cluster. Important If the backup snapshot was taken in an earlier version of Che, add the --version= <earlier_version_number> option to the crwctl server:restore command to roll back Che to that earlier version. Prerequisites Installed crwctl . Backup snapshot of a CodeReady Workspaces instance on any of the following: SFTP backup server Amazon S3 or S3 API compatible backup server REST backup server Internal backup server Backup snapshot ID, which can be found in: Command-line output when creating a backup status of a CheClusterBackup custom object Command-line output of the Restic command that can list snapshots in the backup repository Procedure Run the crwctl server:restore command with the backup snapshot ID specified using the --snapshot-id= option: 1 The snapshot ID. 2 The backup repository URL as an argument using the --repository-url or -r option. (Omit if using the internal backup server!) 3 The backup repository password as an argument using the --repository-password or -p option. (Omit if using the internal backup server!) Tip Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore . When prompted in the output, enter y to confirm: Tip To skip this question, add the --batch option to the server:restore command. Verify the output of the entered command. For example: 8.6.2.3. Restoring a CodeReady Workspaces instance by using crwctl and a CheClusterBackup custom object You can restore a CodeReady Workspaces instance by entering the name of the CheClusterBackup custom object as an argument with the crwctl server:restore command. Important If the backup snapshot was taken in an earlier version of Che, then crwctl server:restore --backup-cr-name= <name_of_custom_object> rolls back Che to that earlier version, which is specified in the CheClusterBackup custom object. Prerequisites Installed crwctl . The CheClusterBackup custom object that was used to create the backup snapshot. The backup snapshot that was taken using the CheClusterBackup custom object and stored on any of the following: SFTP backup server Amazon S3 or S3 API compatible backup server REST backup server Internal backup server Procedure Run the following command on a command line: 1 The name of the CheClusterBackup custom object. When prompted in the output, enter y to confirm: Tip To skip this question, add the --batch option to the server:restore command. Verify the output of the entered command. For example: 8.7. Backups of persistent volumes Persistent Volumes (PVs) store workspace data for CodeReady Workspaces similarly to how workspace data are stored for desktop IDEs on a local hard disk drive. PVs must be backed up periodically to prevent data loss. Storage-agnostic tools are available for backing up and restoring OpenShift resources, including PVs. 8.7.1. Velero An open source tool for backing up OpenShift applications and their PVs is Velero. This tool can perform the following operations: Deploy in the cloud or on premises. Back up the cluster and restore the backed up data in case of data loss. Migrate cluster resources to other clusters. Replicate a production cluster to development and testing clusters. Note Alternatively, you can use backup solutions that depend on the underlying storage system: for example, solutions specific to Gluster or Ceph. Additional resources Kubernetes Documentation - Persistent Volumes Velero Docs - How Velero Works Velero Docs - Disaster Recovery Velero Docs - Backup Reference Velero Docs - Restore Reference Gluster Documentation Ceph Documentation 8.8. Backups of PostgreSQL The CodeReady Workspaces server uses a PostgreSQL database for persisting data about the state of CodeReady Workspaces. The database contains information about user accounts, workspaces, preferences, and other details. 8.8.1. External PostgreSQL setup By default, the CodeReady Workspaces Operator creates and manages deployment of the internal PostgreSQL database. However, the CodeReady Workspaces Operator does not support full lifecycle capabilities such as backups and recovery. A business-critical setup must use an external PostgreSQL database that is configured: for High Availability (HA) and Point In Time Recovery (PITR) on-premises or using a cloud service such as Amazon Relational Database Service (Amazon RDS) For example, Amazon RDS enables deployment of production databases in a Multi-Availability Zone configuration for a resilient disaster recovery strategy with daily and on-demand snapshots. An example configuration is as follows: Parameter Value Instance class db.t2.small vCPU 1 RAM 2 GB Multi-az true, 2 replicas Engine version 9.6.11 TLS enabled Automated backups enabled (30 days) You can make workspace metadata and user information persistent by configuring the external PostgreSQL database and configuring CodeReady Workspaces to use the configured external PostgreSQL database: Section 8.8.2, "Configuring the external PostgreSQL" Section 8.8.3, "Configuring CodeReady Workspaces to work with the external PostgreSQL" Additional resources PostgreSQL Documentation PostgreSQL Documentation - Backup and Restore Amazon RDS 8.8.2. Configuring the external PostgreSQL To make workspace metadata and user information persistent, configure an external PostgreSQL database as follows: Procedure Define the values of the following placeholders: <database-user> CodeReady Workspaces server database user name <database-password> CodeReady Workspaces server database password <database> CodeReady Workspaces server database name Use the following SQL script to create a user and database for the CodeReady Workspaces server to make workspace metadata persistent: CREATE USER <database-user> WITH PASSWORD ' <database-password> ' CREATE DATABASE <database> GRANT ALL PRIVILEGES ON DATABASE <database> TO <database-user> ALTER USER <database-user> WITH SUPERUSER Define the value of the following placeholder: <identity-database-password> RH-SSO database password Use the following SQL script to create a database for the RH-SSO back end to make user information persistent: CREATE USER keycloak WITH PASSWORD ' <identity-database-password> ' CREATE DATABASE keycloak GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak 8.8.3. Configuring CodeReady Workspaces to work with the external PostgreSQL To make workspace metadata and user information persistent, you must configure CodeReady Workspaces to work with the configured external PostgreSQL database. Prerequisites Section 8.8.2, "Configuring the external PostgreSQL" Installed oc Procedure Create a project for CodeReady Workspaces: Create a Secret to store CodeReady Workspaces server database credentials: 1 Secret name to store CodeReady Workspaces server database credentials 2 CodeReady Workspaces server database username 3 CodeReady Workspaces server database password Create a Secret to store RH-SSO database credentials: 1 Secret name to store RH-SSO database credentials 2 RH-SSO database password Deploy Red Hat CodeReady Workspaces by executing the crwctl command and applying a patch. For example: Ensure that patch.yaml contains the following lines to make the Operator skip the deployment of a database and pass connection details of an existing database to a CodeReady Workspaces server: spec: database: externalDb: true chePostgresHostName: <hostname> 1 chePostgresPort: <port> 2 chePostgresSecret: <server-database-credentials> 3 chePostgresDb: <database> 4 spec: auth: identityProviderPostgresSecret: <identity-database-credentials> 5 1 External database hostname 2 External database port 3 Secret name with CodeReady Workspaces server database credentials 4 CodeReady Workspaces server database name 5 Secret name with RH-SSO database credentials
apiVersion: v1 kind: Secret metadata: name: backup-encryption-password-secret namespace: eclipse-che type: Opaque stringData: repo-password: <password> apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration namespace: openshift-workspaces spec: 1 sftp: username: <username> 2 hostname: <hostname> 3 port: 1234 4 repositoryPath: red-hat-codeready-workspaces-backups 5 repositoryPasswordSecretRef: backup-encryption-password-secret 6 sshKeySecretRef: ssh-key-secret 7 apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: red-hat-codeready-workspaces-backup namespace: openshift-workspaces spec: backupServerConfigRef: backup-server-configuration 1 useInternalBackupServer: false 2 oc delete CheClusterBackup <name> -n openshift-workspaces status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 status: message: >- Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC m=+999.553272281 1 snapshotId: 9f0adce2 2 state: Succeeded 3 crwctl server:backup --repository-url= <repository_url> \ 1 --repository-password= <repository_password> \ 2 --ssh-key-file= <ssh_key_file> 3 ✔ Scheduling backup...OK ✔ Waiting until backup process finishes...OK Backup snapshot ID: 9f0adce2 Command server:backup has completed successfully in 00:10. crwctl server:backup --backup-server-config-name= <name_of_CheBackupServerConfiguration> 1 ✔ Scheduling backup...OK ✔ Waiting until backup process finishes...OK Backup snapshot ID: 9f0adce2 Command server:backup has completed successfully in 00:10. apiVersion: v1 kind: Secret metadata: name: backup-encryption-password-secret namespace: eclipse-che type: Opaque stringData: repo-password: <password> apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration namespace: openshift-workspaces spec: 1 awss3: protocol: https 2 hostname: s3.amazonaws.com 3 port: 443 4 repositoryPath: red-hat-codeready-workspaces-backups 5 repositoryPasswordSecretRef: backup-encryption-password-secret 6 awsAccessKeySecretRef: aws-user-credentials-secret 7 apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: red-hat-codeready-workspaces-backup namespace: openshift-workspaces spec: backupServerConfigRef: backup-server-configuration 1 useInternalBackupServer: false 2 oc delete CheClusterBackup <name> -n openshift-workspaces status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 status: message: >- Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC m=+999.553272281 1 snapshotId: 9f0adce2 2 state: Succeeded 3 crwctl server:backup --repository-url= <repository_url> \ 1 --repository-password= <repository_password> \ 2 --aws-access-key-id= <aws-access-key-id> \ 3 --aws-secret-access-key== <aws-secret-access-key> 4 ✔ Scheduling backup...OK ✔ Waiting until backup process finishes...OK Backup snapshot ID: 9f0adce2 Command server:backup has completed successfully in 00:10. crwctl server:backup --backup-server-config-name= <name_of_CheBackupServerConfiguration> 1 ✔ Scheduling backup...OK ✔ Waiting until backup process finishes...OK Backup snapshot ID: 9f0adce2 Command server:backup has completed successfully in 00:10. apiVersion: v1 kind: Secret metadata: name: backup-encryption-password-secret namespace: eclipse-che type: Opaque stringData: repo-password: <password> apiVersion: org.eclipse.che/v1 kind: CheBackupServerConfiguration metadata: name: backup-server-configuration namespace: openshift-workspaces spec: 1 rest: protocol: https 2 hostname: <hostname> 3 port: 1234 4 repositoryPath: red-hat-codeready-workspaces-backups 5 repositoryPasswordSecretRef: backup-encryption-password-secret 6 credentialsSecretRef: rest-server-auth-secret 7 apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: red-hat-codeready-workspaces-backup namespace: openshift-workspaces spec: backupServerConfigRef: backup-server-configuration 1 useInternalBackupServer: false 2 oc delete CheClusterBackup <name> -n openshift-workspaces status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 status: message: >- Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC m=+999.553272281 1 snapshotId: 9f0adce2 2 state: Succeeded 3 crwctl server:backup --repository-url= <repository_url> \ 1 --repository-password= <repository_password> \ 2 --username= <username> \ 3 --password= <password> 4 ✔ Scheduling backup...OK ✔ Waiting until backup process finishes...OK Backup snapshot ID: 9f0adce2 Command server:backup has completed successfully in 00:10. crwctl server:backup --backup-server-config-name= <name_of_CheBackupServerConfiguration> 1 ✔ Scheduling backup...OK ✔ Waiting until backup process finishes...OK Backup snapshot ID: 9f0adce2 Command server:backup has completed successfully in 00:10. apiVersion: org.eclipse.che/v1 kind: CheClusterBackup metadata: name: red-hat-codeready-workspaces-backup namespace: openshift-workspaces spec: useInternalBackupServer: true 1 status: message: 'Backup is in progress. Start time: <timestamp>' 1 stage: Collecting CodeReady Workspaces installation data 2 state: InProgress 3 status: cheVersion: 7.37.0 1 message: >- 2 Backup successfully finished at 2021-10-07 11:18:28.116103737 +0000 UTC m=+2384.430366711 snapshotId: 40a09756 3 state: Succeeded 4 ✔ Scheduling backup...OK ✔ Waiting until backup process finishes...OK Backup snapshot ID: 9f0adce2 Command server:backup has completed successfully in 00:10. apiVersion: org.eclipse.che/v1 kind: CheClusterRestore metadata: name: red-hat-codeready-workspaces-restore namespace: openshift-workspaces spec: backupServerConfigRef: backup-server-configuration 1 snapshotId: <snapshot_id> 2 oc delete CheClusterRestore <name> -n openshift-workspaces status: message: 'Restore is in progress. Start time: <timestamp>' 1 stage: Restoring CodeReady Workspaces related cluster objects 2 state: InProgress 3 crwctl server:restore --snapshot-id=latest \ 1 --repository-url= <repository-url> \ 2 --repository-password= <repository-password> 3 ✔ Snapshot: Asking for restore confirmation: Do you want to proceed? [y/n] ✔ Scheduling restore...OK ✔ Waiting until restore process finishes...OK Command server:restore has completed successfully in 05:59. crwctl server:restore --snapshot-id= <snapshot-id> \ 1 --repository-url= <repository-url> \ 2 --repository-password= <repository-password> 3 ✔ Snapshot: Asking for restore confirmation: Do you want to proceed? [y/n] ✔ Scheduling restore...OK ✔ Waiting until restore process finishes...OK Command server:restore has completed successfully in 05:59. crwctl server:restore --backup-cr-name= <name_of_custom_object> 1 Asking for restore confirmation: Do you want to proceed? [y/n] ✔ Scheduling restore...OK ✔ Waiting until restore process finishes...OK Command server:restore has completed successfully in 05:59. CREATE USER <database-user> WITH PASSWORD ' <database-password> ' CREATE DATABASE <database> GRANT ALL PRIVILEGES ON DATABASE <database> TO <database-user> ALTER USER <database-user> WITH SUPERUSER CREATE USER keycloak WITH PASSWORD ' <identity-database-password> ' CREATE DATABASE keycloak GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak oc create namespace openshift-workspaces oc create secret generic <server-database-credentials> \ 1 --from-literal=user= <database-user> \ 2 --from-literal=password= <database-password> \ 3 -n openshift-workspaces oc create secret generic <identity-database-credentials> \ 1 --from-literal=password= <identity-database-password> \ 2 -n openshift-workspaces crwctl server:deploy --che-operator-cr-patch-yaml=patch.yaml spec: database: externalDb: true chePostgresHostName: <hostname> 1 chePostgresPort: <port> 2 chePostgresSecret: <server-database-credentials> 3 chePostgresDb: <database> 4 spec: auth: identityProviderPostgresSecret: <identity-database-credentials> 5