Install ROSA with HCP clusters
Install ROSA with HCP clusters Red Hat OpenShift Service on AWS 4 Installing, accessing, and deleting Red Hat OpenShift Service on AWS (ROSA) clusters. Red Hat OpenShift Documentation Team
git clone https://github.com/openshift-cs/terraform-vpc-example cd terraform-vpc-example terraform init terraform plan -out rosa.tfplan -var region=<region> terraform apply rosa.tfplan export SUBNET_IDS=USD(terraform output -raw cluster-subnets-string) echo USDSUBNET_IDS subnet-0a6a57e0f784171aa,subnet-078e84e5b10ecf5b0 aws ec2 create-tags --resources <public-subnet-id> --region <aws_region> --tags Key=kubernetes.io/role/elb,Value=1 aws ec2 create-tags --resources <private-subnet-id> --region <aws_region> --tags Key=kubernetes.io/role/internal-elb,Value=1 aws ec2 describe-tags --filters "Name=resource-id,Values=<subnet_id>" TAGS Name <subnet-id> subnet <prefix>-subnet-public1-us-east-1a TAGS kubernetes.io/role/elb <subnet-id> subnet 1 rosa create account-roles --hosted-cp export ACCOUNT_ROLES_PREFIX=<account_role_prefix> echo USDACCOUNT_ROLES_PREFIX ManagedOpenShift rosa create oidc-config --mode=auto --yes ? Would you like to create a Managed (Red Hat hosted) OIDC Configuration Yes I: Setting up managed OIDC configuration I: To create Operator Roles for this OIDC Configuration, run the following command and remember to replace <user-defined> with a prefix of your choice: rosa create operator-roles --prefix <user-defined> --oidc-config-id 13cdr6b If you are going to create a Hosted Control Plane cluster please include '--hosted-cp' I: Creating OIDC provider using 'arn:aws:iam::4540112244:user/userName' ? Create the OIDC provider? Yes I: Created OIDC provider with ARN 'arn:aws:iam::4540112244:oidc-provider/dvbwgdztaeq9o.cloudfront.net/13cdr6b' export OIDC_ID=<oidc_config_id> 1 echo USDOIDC_ID 13cdr6b rosa list oidc-config ID MANAGED ISSUER URL SECRET ARN 2330dbs0n8m3chkkr25gkkcd8pnj3lk2 true https://dvbwgdztaeq9o.cloudfront.net/2330dbs0n8m3chkkr25gkkcd8pnj3lk2 233hvnrjoqu14jltk6lhbhf2tj11f8un false https://oidc-r7u1.s3.us-east-1.amazonaws.com aws:secretsmanager:us-east-1:242819244:secret:rosa-private-key-oidc-r7u1-tM3MDN export OPERATOR_ROLES_PREFIX=<prefix_name> rosa create operator-roles --hosted-cp --prefix=USDOPERATOR_ROLES_PREFIX --oidc-config-id=USDOIDC_ID --installer-role-arn arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/USD{ACCOUNT_ROLES_PREFIX}-HCP-ROSA-Installer-Role rosa create operator-roles --hosted-cp --prefix=USDOPERATOR_ROLES_PREFIX 1 --oidc-config-id=USDOIDC_ID 2 --installer-role-arn arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/USD{ACCOUNT_ROLES_PREFIX}-HCP-ROSA-Installer-Role 3 ? Role creation mode: auto ? Operator roles prefix: <pre-filled_prefix> 1 ? OIDC Configuration ID: 23soa2bgvpek9kmes9s7os0a39i13qm4 | https://dvbwgdztaeq9o.cloudfront.net/23soa2bgvpek9kmes9s7os0a39i13qm4 2 ? Create hosted control plane operator roles: Yes W: More than one Installer role found ? Installer role ARN: arn:aws:iam::4540112244:role/<prefix>-HCP-ROSA-Installer-Role ? Permissions boundary ARN (optional): I: Reusable OIDC Configuration detected. Validating trusted relationships to operator roles: I: Creating roles using 'arn:aws:iam::4540112244:user/<userName>' I: Created role '<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials' I: Created role '<prefix>-openshift-cloud-network-config-controller-cloud-credenti' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-cloud-network-config-controller-cloud-credenti' I: Created role '<prefix>-kube-system-kube-controller-manager' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-kube-controller-manager' I: Created role '<prefix>-kube-system-capa-controller-manager' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-capa-controller-manager' I: Created role '<prefix>-kube-system-control-plane-operator' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-control-plane-operator' I: Created role '<prefix>-kube-system-kms-provider' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-kms-provider' I: Created role '<prefix>-openshift-image-registry-installer-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-image-registry-installer-cloud-credentials' I: Created role '<prefix>-openshift-ingress-operator-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-ingress-operator-cloud-credentials' I: To create a cluster with these roles, run the following command: rosa create cluster --sts --oidc-config-id 23soa2bgvpek9kmes9s7os0a39i13qm4 --operator-roles-prefix <prefix> --hosted-cp rosa list operator-roles I: Fetching operator roles ROLE PREFIX AMOUNT IN BUNDLE <prefix> 8 ? Would you like to detail a specific prefix Yes 1 ? Operator Role Prefix: <prefix> ROLE NAME ROLE ARN VERSION MANAGED <prefix>-kube-system-capa-controller-manager arn:aws:iam::4540112244:role/<prefix>-kube-system-capa-controller-manager 4.13 No <prefix>-kube-system-control-plane-operator arn:aws:iam::4540112244:role/<prefix>-kube-system-control-plane-operator 4.13 No <prefix>-kube-system-kms-provider arn:aws:iam::4540112244:role/<prefix>-kube-system-kms-provider 4.13 No <prefix>-kube-system-kube-controller-manager arn:aws:iam::4540112244:role/<prefix>-kube-system-kube-controller-manager 4.13 No <prefix>-openshift-cloud-network-config-controller-cloud-credenti arn:aws:iam::4540112244:role/<prefix>-openshift-cloud-network-config-controller-cloud-credenti 4.13 No <prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials 4.13 No <prefix>-openshift-image-registry-installer-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-image-registry-installer-cloud-credentials 4.13 No <prefix>-openshift-ingress-operator-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-ingress-operator-cloud-credentials 4.13 No rosa create cluster --cluster-name=<cluster_name> \ 1 --mode=auto --hosted-cp [--private] \ 2 --operator-roles-prefix <operator-role-prefix> \ 3 --external-id <external-id> \ 4 --oidc-config-id <id-of-oidc-configuration> --subnet-ids=<public-subnet-id>,<private-subnet-id> rosa create cluster --private --cluster-name=<cluster_name> --mode=auto --hosted-cp --operator-roles-prefix=USDOPERATOR_ROLES_PREFIX --oidc-config-id=USDOIDC_ID --subnet-ids=USDSUBNET_IDS rosa create cluster --cluster-name=<cluster_name> --mode=auto --hosted-cp --operator-roles-prefix=USDOPERATOR_ROLES_PREFIX --oidc-config-id=USDOIDC_ID --subnet-ids=USDSUBNET_IDS rosa describe cluster --cluster=<cluster_name> rosa logs install --cluster=<cluster_name> --watch \ <.> { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "iam:GetPolicyVersion", "iam:DeletePolicyVersion", "iam:CreatePolicyVersion", "iam:UpdateAssumeRolePolicy", "secretsmanager:DescribeSecret", "iam:ListRoleTags", "secretsmanager:PutSecretValue", "secretsmanager:CreateSecret", "iam:TagRole", "secretsmanager:DeleteSecret", "iam:UpdateOpenIDConnectProviderThumbprint", "iam:DeletePolicy", "iam:CreateRole", "iam:AttachRolePolicy", "iam:ListInstanceProfilesForRole", "secretsmanager:GetSecretValue", "iam:DetachRolePolicy", "iam:ListAttachedRolePolicies", "iam:ListPolicyTags", "iam:ListRolePolicies", "iam:DeleteOpenIDConnectProvider", "iam:DeleteInstanceProfile", "iam:GetRole", "iam:GetPolicy", "iam:ListEntitiesForPolicy", "iam:DeleteRole", "iam:TagPolicy", "iam:CreateOpenIDConnectProvider", "iam:CreatePolicy", "secretsmanager:GetResourcePolicy", "iam:ListPolicyVersions", "iam:UpdateRole", "iam:GetOpenIDConnectProvider", "iam:TagOpenIDConnectProvider", "secretsmanager:TagResource", "sts:AssumeRoleWithWebIdentity", "iam:ListRoles" ], "Resource": [ "arn:aws:secretsmanager:*:<ACCOUNT_ID>:secret:*", "arn:aws:iam::<ACCOUNT_ID>:instance-profile/*", "arn:aws:iam::<ACCOUNT_ID>:role/*", "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/*", "arn:aws:iam::<ACCOUNT_ID>:policy/*" ] }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "s3:*" ], "Resource": "*" } ] } mkdir terraform-cluster && cd terraform-cluster export RHCS_TOKEN=<your_offline_token> echo USDRHCS_TOKEN cat<<-EOF>main.tf # Copyright (c) 2023 Red Hat, Inc. # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. # terraform { required_providers { aws = { source = "hashicorp/aws" version = ">= 4.20.0" } rhcs = { version = ">= 1.6.3" source = "terraform-redhat/rhcs" } } } Export token using the RHCS_TOKEN environment variable provider "rhcs" {} provider "aws" { region = var.aws_region ignore_tags { key_prefixes = ["kubernetes.io/"] } default_tags { tags = var.default_aws_tags } } data "aws_availability_zones" "available" {} locals { # Extract availability zone names for the specified region, limit it to 3 if multi az or 1 if single region_azs = var.multi_az ? slice([for zone in data.aws_availability_zones.available.names : format("%s", zone)], 0, 3) : slice([for zone in data.aws_availability_zones.available.names : format("%s", zone)], 0, 1) } resource "random_string" "random_name" { length = 6 special = false upper = false } locals { worker_node_replicas = var.multi_az ? 3 : 2 # If cluster_name is not null, use that, otherwise generate a random cluster name cluster_name = coalesce(var.cluster_name, "rosa-\USD{random_string.random_name.result}") } The network validator requires an additional 60 seconds to validate Terraform clusters. resource "time_sleep" "wait_60_seconds" { count = var.create_vpc ? 1 : 0 depends_on = [module.vpc] create_duration = "60s" } module "rosa-hcp" { source = "terraform-redhat/rosa-hcp/rhcs" version = "1.6.3" cluster_name = local.cluster_name openshift_version = var.openshift_version account_role_prefix = local.cluster_name operator_role_prefix = local.cluster_name replicas = local.worker_node_replicas aws_availability_zones = local.region_azs create_oidc = true private = var.private_cluster aws_subnet_ids = var.create_vpc ? var.private_cluster ? module.vpc[0].private_subnets : concat(module.vpc[0].public_subnets, module.vpc[0].private_subnets) : var.aws_subnet_ids create_account_roles = true create_operator_roles = true Optional: Configure a cluster administrator user <.> # Option 1: Default cluster-admin user Create an administrator user (cluster-admin) and automatically generate a password by uncommenting the following parameter: create_admin_user = true Generated administrator credentials are displayed in terminal output. # Option 2: Specify administrator username and password Create an administrator user and define your own password by uncommenting and editing the values of the following parameters: admin_credentials_username = <username> admin_credentials_password = <password> depends_on = [time_sleep.wait_60_seconds] } EOF cat<<-EOF>variables.tf # Copyright (c) 2023 Red Hat, Inc. # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. # variable "openshift_version" { type = string default = "4.14.20" description = "Desired version of OpenShift for the cluster, for example '4.14.20'. If version is greater than the currently running version, an upgrade will be scheduled." } variable "create_vpc" { type = bool description = "If you would like to create a new VPC, set this value to 'true'. If you do not want to create a new VPC, set this value to 'false'." } ROSA Cluster info variable "cluster_name" { default = null type = string description = "The name of the ROSA cluster to create" } variable "additional_tags" { default = { Terraform = "true" Environment = "dev" } description = "Additional AWS resource tags" type = map(string) } variable "multi_az" { type = bool description = "Multi AZ Cluster for High Availability" default = true } variable "worker_node_replicas" { default = 3 description = "Number of worker nodes to provision. Single zone clusters need at least 2 nodes, multizone clusters need at least 3 nodes" type = number } variable "aws_subnet_ids" { type = list(any) description = "A list of either the public or public + private subnet IDs to use for the cluster blocks to use for the cluster" default = ["subnet-01234567890abcdef", "subnet-01234567890abcdef", "subnet-01234567890abcdef"] } variable "private_cluster" { type = bool description = "If you want to create a private cluster, set this value to 'true'. If you want a publicly available cluster, set this value to 'false'." } #VPC Info variable "vpc_name" { type = string description = "VPC Name" default = "tf-qs-vpc" } variable "vpc_cidr_block" { type = string description = "value of the CIDR block to use for the VPC" default = "10.0.0.0/16" } variable "private_subnet_cidrs" { type = list(any) description = "The CIDR blocks to use for the private subnets" default = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"] } variable "public_subnet_cidrs" { type = list(any) description = "The CIDR blocks to use for the public subnets" default = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"] } variable "single_nat_gateway" { type = bool description = "Single NAT or per NAT for subnet" default = false } #AWS Info variable "aws_region" { type = string default = "us-east-2" } variable "default_aws_tags" { type = map(string) description = "Default tags for AWS" default = {} } EOF cat<<-EOF>vpc.tf # Copyright (c) 2023 Red Hat, Inc. # Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. # module "vpc" { source = "terraform-aws-modules/vpc/aws" version = "5.1.2" count = var.create_vpc ? 1 : 0 name = var.vpc_name cidr = var.vpc_cidr_block azs = local.region_azs private_subnets = var.multi_az ? var.private_subnet_cidrs : [var.private_subnet_cidrs[0]] public_subnets = var.multi_az ? var.public_subnet_cidrs : [var.public_subnet_cidrs[0]] enable_nat_gateway = true single_nat_gateway = var.single_nat_gateway enable_dns_hostnames = true enable_dns_support = true tags = var.additional_tags } EOF terraform init terraform validate Success! The configuration is valid. terraform apply var.create_vpc If you would like to create a new VPC, set this value to 'true'. If you do not want to create a new VPC, set this value to 'false'. Enter a value: var.private_cluster If you want to create a private cluster, set this value to 'true'. If you want a publicly available cluster, set this value to 'false'. Enter a value: Plan: 63 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. rosa list clusters ID NAME STATE TOPOLOGY 27c3snjsupa9obua74ba8se5kcj11269 rosa-tf-demo ready Classic (STS) rosa list account-roles I: Fetching account roles ROLE NAME ROLE TYPE ROLE ARN OPENSHIFT VERSION AWS Managed ROSA-demo-Installer-Role Installer arn:aws:iam::<ID>:role/ROSA-demo-Installer-Role 4.14 No ROSA-demo-Support-Role Support arn:aws:iam::<ID>:role/ROSA-demo-Support-Role 4.14 No ROSA-demo-Worker-Role Worker arn:aws:iam::<ID>:role/ROSA-demo-Worker-Role 4.14 No rosa list operator-roles I: Fetching operator roles ROLE PREFIX AMOUNT IN BUNDLE rosa-demo 8 terraform destroy var.create_vpc If you would like to create a new VPC, set this value to 'true.' If you do not want to create a new VPC, set this value to 'false.' Enter a value: var.private_cluster If you want to create a private cluster, set this value to 'true.' If you want a publicly available cluster, set this value to 'false.' Enter a value: Plan: 0 to add, 0 to change, 63 to destroy. Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes rosa list clusters I: No clusters available rosa list account-roles I: Fetching account roles I: No account roles available rosa list operator-roles I: Fetching operator roles I: No operator roles available git clone https://github.com/openshift-cs/terraform-vpc-example cd terraform-vpc-example terraform init terraform plan -out rosa.tfplan -var region=<region> terraform apply rosa.tfplan export SUBNET_IDS=USD(terraform output -raw cluster-subnets-string) echo USDSUBNET_IDS subnet-0a6a57e0f784171aa,subnet-078e84e5b10ecf5b0 rosa create account-roles --hosted-cp export ACCOUNT_ROLES_PREFIX=<account_role_prefix> echo USDACCOUNT_ROLES_PREFIX ManagedOpenShift rosa create oidc-config --mode=auto --yes ? Would you like to create a Managed (Red Hat hosted) OIDC Configuration Yes I: Setting up managed OIDC configuration I: To create Operator Roles for this OIDC Configuration, run the following command and remember to replace <user-defined> with a prefix of your choice: rosa create operator-roles --prefix <user-defined> --oidc-config-id 13cdr6b If you are going to create a Hosted Control Plane cluster please include '--hosted-cp' I: Creating OIDC provider using 'arn:aws:iam::4540112244:user/userName' ? Create the OIDC provider? Yes I: Created OIDC provider with ARN 'arn:aws:iam::4540112244:oidc-provider/dvbwgdztaeq9o.cloudfront.net/13cdr6b' export OIDC_ID=<oidc_config_id> 1 echo USDOIDC_ID 13cdr6b rosa list oidc-config ID MANAGED ISSUER URL SECRET ARN 2330dbs0n8m3chkkr25gkkcd8pnj3lk2 true https://dvbwgdztaeq9o.cloudfront.net/2330dbs0n8m3chkkr25gkkcd8pnj3lk2 233hvnrjoqu14jltk6lhbhf2tj11f8un false https://oidc-r7u1.s3.us-east-1.amazonaws.com aws:secretsmanager:us-east-1:242819244:secret:rosa-private-key-oidc-r7u1-tM3MDN export OPERATOR_ROLES_PREFIX=<prefix_name> rosa create operator-roles --hosted-cp --prefix=USDOPERATOR_ROLES_PREFIX --oidc-config-id=USDOIDC_ID --installer-role-arn arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/USD{ACCOUNT_ROLES_PREFIX}-HCP-ROSA-Installer-Role rosa create operator-roles --hosted-cp --prefix=USDOPERATOR_ROLES_PREFIX 1 --oidc-config-id=USDOIDC_ID 2 --installer-role-arn arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/USD{ACCOUNT_ROLES_PREFIX}-HCP-ROSA-Installer-Role 3 ? Role creation mode: auto ? Operator roles prefix: <pre-filled_prefix> 1 ? OIDC Configuration ID: 23soa2bgvpek9kmes9s7os0a39i13qm4 | https://dvbwgdztaeq9o.cloudfront.net/23soa2bgvpek9kmes9s7os0a39i13qm4 2 ? Create hosted control plane operator roles: Yes W: More than one Installer role found ? Installer role ARN: arn:aws:iam::4540112244:role/<prefix>-HCP-ROSA-Installer-Role ? Permissions boundary ARN (optional): I: Reusable OIDC Configuration detected. Validating trusted relationships to operator roles: I: Creating roles using 'arn:aws:iam::4540112244:user/<userName>' I: Created role '<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials' I: Created role '<prefix>-openshift-cloud-network-config-controller-cloud-credenti' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-cloud-network-config-controller-cloud-credenti' I: Created role '<prefix>-kube-system-kube-controller-manager' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-kube-controller-manager' I: Created role '<prefix>-kube-system-capa-controller-manager' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-capa-controller-manager' I: Created role '<prefix>-kube-system-control-plane-operator' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-control-plane-operator' I: Created role '<prefix>-kube-system-kms-provider' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-kms-provider' I: Created role '<prefix>-openshift-image-registry-installer-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-image-registry-installer-cloud-credentials' I: Created role '<prefix>-openshift-ingress-operator-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-ingress-operator-cloud-credentials' I: To create a cluster with these roles, run the following command: rosa create cluster --sts --oidc-config-id 23soa2bgvpek9kmes9s7os0a39i13qm4 --operator-roles-prefix <prefix> --hosted-cp rosa list operator-roles I: Fetching operator roles ROLE PREFIX AMOUNT IN BUNDLE <prefix> 8 ? Would you like to detail a specific prefix Yes 1 ? Operator Role Prefix: <prefix> ROLE NAME ROLE ARN VERSION MANAGED <prefix>-kube-system-capa-controller-manager arn:aws:iam::4540112244:role/<prefix>-kube-system-capa-controller-manager 4.13 No <prefix>-kube-system-control-plane-operator arn:aws:iam::4540112244:role/<prefix>-kube-system-control-plane-operator 4.13 No <prefix>-kube-system-kms-provider arn:aws:iam::4540112244:role/<prefix>-kube-system-kms-provider 4.13 No <prefix>-kube-system-kube-controller-manager arn:aws:iam::4540112244:role/<prefix>-kube-system-kube-controller-manager 4.13 No <prefix>-openshift-cloud-network-config-controller-cloud-credenti arn:aws:iam::4540112244:role/<prefix>-openshift-cloud-network-config-controller-cloud-credenti 4.13 No <prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials 4.13 No <prefix>-openshift-image-registry-installer-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-image-registry-installer-cloud-credentials 4.13 No <prefix>-openshift-ingress-operator-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-ingress-operator-cloud-credentials 4.13 No KMS_ARN=USD(aws kms create-key --region USDAWS_REGION --description 'Custom ROSA Encryption Key' --tags TagKey=red-hat,TagValue=true --query KeyMetadata.Arn --output text) echo USDKMS_ARN AWS_ACCOUNT_ID=<aws_account_id> { "Version": "2012-10-17", "Id": "key-rosa-policy-1", "Statement": [ { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::USD{AWS_ACCOUNT_ID}:root" }, "Action": "kms:*", "Resource": "*" }, { "Sid": "Installer Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/ManagedOpenShift-HCP-ROSA-Installer-Role" }, "Action": [ "kms:CreateGrant", "kms:DescribeKey", "kms:GenerateDataKeyWithoutPlaintext" ], "Resource": "*" }, { "Sid": "ROSA KubeControllerManager Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/<operator_role_prefix>-kube-system-kube-controller-manager" }, "Action": "kms:DescribeKey", "Resource": "*" }, { "Sid": "ROSA KMS Provider Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/<operator_role_prefix>-kube-system-kms-provider" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:DescribeKey" ], "Resource": "*" }, { "Sid": "ROSA NodeManager Permissions", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/<operator_role_prefix>-kube-system-capa-controller-manager" }, "Action": [ "kms:DescribeKey", "kms:GenerateDataKeyWithoutPlaintext", "kms:CreateGrant" ], "Resource": "*" } ] } cat rosa-key-policy.json aws kms put-key-policy --key-id USDKMS_ARN --policy file://rosa-key-policy.json --policy-name default rosa create cluster --cluster-name <cluster_name> --subnet-ids <private_subnet_id>,<public_subnet_id> --sts --mode auto --machine-cidr 10.0.0.0/16 --compute-machine-type m5.xlarge --hosted-cp --region <aws_region> --oidc-config-id USDOIDC_ID --kms-key-arn USDKMS_ARN \ 1 --etcd-encryption-kms-arn USDKMS_ARN \ 2 --operator-roles-prefix USDOPERATOR_ROLES_PREFIX rosa create account-roles --hosted-cp rosa create oidc-config --mode=auto --yes I: Setting up managed OIDC configuration I: To create Operator Roles for this OIDC Configuration, run the following command and remember to replace <user-defined> with a prefix of your choice: rosa create operator-roles --prefix <user-defined> --oidc-config-id 28s4avcdt2l318r1jbk3ifmimkurk384 If you are going to create a Hosted Control Plane cluster please include '--hosted-cp' I: Creating OIDC provider using 'arn:aws:iam::46545644412:user/user' I: Created OIDC provider with ARN 'arn:aws:iam::46545644412:oidc-provider/oidc.op1.openshiftapps.com/28s4avcdt2l318r1jbk3ifmimkurk384' rosa create operator-roles --hosted-cp --prefix <operator_roles_prefix> --oidc-config-id <oidc_config_id> --installer-role-arn arn:aws:iam::USD<account_roles_prefix>:role/USD<account_roles_prefix>-HCP-ROSA-Installer-Role rosa create cluster --private --cluster-name=<cluster-name> --sts --mode=auto --hosted-cp --operator-roles-prefix <operator_role_prefix> --oidc-config-id <oidc_config_id> [--machine-cidr=<VPC CIDR>/16] --subnet-ids=<private-subnet-id1>[,<private-subnet-id2>,<private-subnet-id3>] rosa describe cluster --cluster=<cluster_name> rosa logs install --cluster=<cluster_name> --watch export CLUSTER_NAME=<cluster_name> echo USDCLUSTER_NAME hcp-private read -r VPCE_ID VPC_ID <<< USD(aws ec2 describe-vpc-endpoints --filters "Name=tag:api.openshift.com/id,Values=USD(rosa describe cluster -c USD{CLUSTER_NAME} -o yaml | grep '^id: ' | cut -d' ' -f2)" --query 'VpcEndpoints[].[VpcEndpointId,VpcId]' --output text) export SG_ID=USD(aws ec2 create-security-group --description "Granting API access to USD{CLUSTER_NAME} from outside of VPC" --group-name "USD{CLUSTER_NAME}-api-sg" --vpc-id USDVPC_ID --output text) aws ec2 authorize-security-group-ingress --group-id USDSG_ID --ip-permissions FromPort=443,ToPort=443,IpProtocol=tcp,IpRanges=[{CidrIp=<cidr-to-allow>}] <.> aws ec2 modify-vpc-endpoint --vpc-endpoint-id USDVPCE_ID --add-security-group-ids USDSG_ID rosa create cluster [...] --additional-allowed-principals <arn_string> Name: mycluster Domain Prefix: mycluster Display Name: mycluster ID: <cluster-id> External ID: <cluster-id> Control Plane: ROSA Service Hosted OpenShift Version: 4.15.17 Channel Group: stable DNS: Not ready AWS Account: <aws_id> AWS Billing Account: <aws_id> API URL: Console URL: Region: us-east-2 Availability: - Control Plane: MultiAZ - Data Plane: SingleAZ Nodes: - Compute (desired): 2 - Compute (current): 0 Network: - Type: OVNKubernetes - Service CIDR: 172.30.0.0/16 - Machine CIDR: 10.0.0.0/16 - Pod CIDR: 10.128.0.0/14 - Host Prefix: /23 - Subnets: subnet-453e99d40, subnet-666847ce827 EC2 Metadata Http Tokens: optional Role (STS) ARN: arn:aws:iam::<aws_id>:role/mycluster-HCP-ROSA-Installer-Role Support Role ARN: arn:aws:iam::<aws_id>:role/mycluster-HCP-ROSA-Support-Role Instance IAM Roles: - Worker: arn:aws:iam::<aws_id>:role/mycluster-HCP-ROSA-Worker-Role Operator IAM Roles: - arn:aws:iam::<aws_id>:role/mycluster-kube-system-control-plane-operator - arn:aws:iam::<aws_id>:role/mycluster-openshift-cloud-network-config-controller-cloud-creden - arn:aws:iam::<aws_id>:role/mycluster-openshift-image-registry-installer-cloud-credentials - arn:aws:iam::<aws_id>:role/mycluster-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<aws_id>:role/mycluster-openshift-cluster-csi-drivers-ebs-cloud-credentials - arn:aws:iam::<aws_id>:role/mycluster-kube-system-kms-provider - arn:aws:iam::<aws_id>:role/mycluster-kube-system-kube-controller-manager - arn:aws:iam::<aws_id>:role/mycluster-kube-system-capa-controller-manager Managed Policies: Yes State: waiting (Waiting for user action) Private: No Delete Protection: Disabled Created: Jun 25 2024 13:36:37 UTC User Workload Monitoring: Enabled Details Page: https://console.redhat.com/openshift/details/s/Bvbok4O79q1Vg8 OIDC Endpoint URL: https://oidc.op1.openshiftapps.com/vhufi5lap6vbl3jlq20e (Managed) Audit Log Forwarding: Disabled External Authentication: Disabled Additional Principals: arn:aws:iam::<aws_id>:role/additional-user-role rosa edit cluster -c <cluster_name> --additional-allowed-principals <arn_string> git clone https://github.com/openshift-cs/terraform-vpc-example cd terraform-vpc-example/zero-egress terraform init terraform plan -out rosa-zero-egress.tfplan -var region=<aws_region> \ 1 -var 'availability_zones=["aws_region_1a","aws_region_1b","aws_region_1c"]'\ 2 -var vpc_cidr_block=10.0.0.0/16 \ 3 -var 'private_subnets=["10.0.0.0/24", "10.0.1.0/24", "10.0.2.0/24"]' 4 terraform apply rosa-zero-egress.tfplan aws ec2 create-tags --resources <public-subnet-id> --region <aws_region> --tags Key=kubernetes.io/role/elb,Value=1 aws ec2 create-tags --resources <private-subnet-id> --region <aws_region> --tags Key=kubernetes.io/role/internal-elb,Value=1 aws ec2 describe-tags --filters "Name=resource-id,Values=<subnet_id>" TAGS Name <subnet-id> subnet <prefix>-subnet-public1-us-east-1a TAGS kubernetes.io/role/elb <subnet-id> subnet 1 aws ec2 create-security-group --group-name allow-inbound-traffic --description "allow inbound traffic" --vpc-id <vpc_id> \ 1 --region <aws_region> \ 2 aws ec2 authorize-security-group-ingress --group-id <group_id> \ 1 --protocol -1 --port 0-0 --cidr <vpc_cidr> \ 2 --region <aws_region> \ 3 aws ec2 create-vpc-endpoint --vpc-id <vpc_id> \ 1 --service-name com.amazonaws.<aws_region>.sts \ 2 --vpc-endpoint-type Interface aws ec2 create-vpc-endpoint --vpc-id <vpc_id> --service-name com.amazonaws.<aws_region>.ecr.dkr \ 1 --vpc-endpoint-type Interface aws ec2 create-vpc-endpoint --vpc-id <vpc_id> --service-name com.amazonaws.<aws_region>.s3 rosa create account-roles --hosted-cp aws iam attach-role-policy --role-name ManagedOpenShift-HCP-ROSA-Worker-Role \ 1 --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" export ACCOUNT_ROLES_PREFIX=<account_role_prefix> echo USDACCOUNT_ROLES_PREFIX ManagedOpenShift rosa create oidc-config --mode=auto --yes ? Would you like to create a Managed (Red Hat hosted) OIDC Configuration Yes I: Setting up managed OIDC configuration I: To create Operator Roles for this OIDC Configuration, run the following command and remember to replace <user-defined> with a prefix of your choice: rosa create operator-roles --prefix <user-defined> --oidc-config-id 13cdr6b If you are going to create a Hosted Control Plane cluster please include '--hosted-cp' I: Creating OIDC provider using 'arn:aws:iam::4540112244:user/userName' ? Create the OIDC provider? Yes I: Created OIDC provider with ARN 'arn:aws:iam::4540112244:oidc-provider/dvbwgdztaeq9o.cloudfront.net/13cdr6b' export OIDC_ID=<oidc_config_id> 1 echo USDOIDC_ID 13cdr6b rosa list oidc-config ID MANAGED ISSUER URL SECRET ARN 2330dbs0n8m3chkkr25gkkcd8pnj3lk2 true https://dvbwgdztaeq9o.cloudfront.net/2330dbs0n8m3chkkr25gkkcd8pnj3lk2 233hvnrjoqu14jltk6lhbhf2tj11f8un false https://oidc-r7u1.s3.us-east-1.amazonaws.com aws:secretsmanager:us-east-1:242819244:secret:rosa-private-key-oidc-r7u1-tM3MDN export OPERATOR_ROLES_PREFIX=<prefix_name> rosa create operator-roles --hosted-cp --prefix=USDOPERATOR_ROLES_PREFIX --oidc-config-id=USDOIDC_ID --installer-role-arn arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/USD{ACCOUNT_ROLES_PREFIX}-HCP-ROSA-Installer-Role rosa create operator-roles --hosted-cp --prefix=USDOPERATOR_ROLES_PREFIX 1 --oidc-config-id=USDOIDC_ID 2 --installer-role-arn arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/USD{ACCOUNT_ROLES_PREFIX}-HCP-ROSA-Installer-Role 3 ? Role creation mode: auto ? Operator roles prefix: <pre-filled_prefix> 1 ? OIDC Configuration ID: 23soa2bgvpek9kmes9s7os0a39i13qm4 | https://dvbwgdztaeq9o.cloudfront.net/23soa2bgvpek9kmes9s7os0a39i13qm4 2 ? Create hosted control plane operator roles: Yes W: More than one Installer role found ? Installer role ARN: arn:aws:iam::4540112244:role/<prefix>-HCP-ROSA-Installer-Role ? Permissions boundary ARN (optional): I: Reusable OIDC Configuration detected. Validating trusted relationships to operator roles: I: Creating roles using 'arn:aws:iam::4540112244:user/<userName>' I: Created role '<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials' I: Created role '<prefix>-openshift-cloud-network-config-controller-cloud-credenti' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-cloud-network-config-controller-cloud-credenti' I: Created role '<prefix>-kube-system-kube-controller-manager' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-kube-controller-manager' I: Created role '<prefix>-kube-system-capa-controller-manager' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-capa-controller-manager' I: Created role '<prefix>-kube-system-control-plane-operator' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-control-plane-operator' I: Created role '<prefix>-kube-system-kms-provider' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-kms-provider' I: Created role '<prefix>-openshift-image-registry-installer-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-image-registry-installer-cloud-credentials' I: Created role '<prefix>-openshift-ingress-operator-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-ingress-operator-cloud-credentials' I: To create a cluster with these roles, run the following command: rosa create cluster --sts --oidc-config-id 23soa2bgvpek9kmes9s7os0a39i13qm4 --operator-roles-prefix <prefix> --hosted-cp rosa list operator-roles I: Fetching operator roles ROLE PREFIX AMOUNT IN BUNDLE <prefix> 8 ? Would you like to detail a specific prefix Yes 1 ? Operator Role Prefix: <prefix> ROLE NAME ROLE ARN VERSION MANAGED <prefix>-kube-system-capa-controller-manager arn:aws:iam::4540112244:role/<prefix>-kube-system-capa-controller-manager 4.13 No <prefix>-kube-system-control-plane-operator arn:aws:iam::4540112244:role/<prefix>-kube-system-control-plane-operator 4.13 No <prefix>-kube-system-kms-provider arn:aws:iam::4540112244:role/<prefix>-kube-system-kms-provider 4.13 No <prefix>-kube-system-kube-controller-manager arn:aws:iam::4540112244:role/<prefix>-kube-system-kube-controller-manager 4.13 No <prefix>-openshift-cloud-network-config-controller-cloud-credenti arn:aws:iam::4540112244:role/<prefix>-openshift-cloud-network-config-controller-cloud-credenti 4.13 No <prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials 4.13 No <prefix>-openshift-image-registry-installer-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-image-registry-installer-cloud-credentials 4.13 No <prefix>-openshift-ingress-operator-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-ingress-operator-cloud-credentials 4.13 No rosa create cluster --cluster-name=<cluster_name> \ <.> --mode=auto --hosted-cp [--private] \ <.> --operator-roles-prefix <operator-role-prefix> \ <.> --oidc-config-id <id-of-oidc-configuration> --subnet-ids=<private-subnet-id> --region <region> --machine-cidr 10.0.0.0/16 --service-cidr 172.30.0.0/16 --pod-cidr 10.128.0.0/14 --host-prefix 23 --billing-account <root-acct-id> \ <.> --properties zero_egress:true rosa create cluster --private --cluster-name=<cluster_name> --mode=auto --hosted-cp --operator-roles-prefix=USDOPERATOR_ROLES_PREFIX --oidc-config-id=USDOIDC_ID --subnet-ids=USDSUBNET_IDS --region <region> --machine-cidr 10.0.0.0/16 --service-cidr 172.30.0.0/16 --pod-cidr 10.128.0.0/14 --host-prefix 23 --billing-account <root-acct-id> --private --properties zero_egress:true rosa describe cluster --cluster=<cluster_name> rosa logs install --cluster=<cluster_name> --watch \ <.> rosa create cluster --hosted-cp --subnet-ids=USDSUBNET_IDS --oidc-config-id=USDOIDC_ID --cluster-name=<cluster_name> --operator-roles-prefix=USDOPERATOR_ROLES_PREFIX --external-auth-providers-enabled rosa create cluster --cluster-name=<cluster_name> --sts --mode=auto --hosted-cp --operator-roles-prefix <operator-role-prefix> --oidc-config-id <ID-of-OIDC-configuration> --external-auth-providers-enabled --subnet-ids=<public-subnet-id>,<private-subnet-id> rosa describe cluster --cluster=<cluster_name> Name: rosa-ext-test Display Name: rosa-ext-test ID: <cluster_id> External ID: <cluster_ext_id> Control Plane: ROSA Service Hosted OpenShift Version: 4.18.0 Channel Group: stable DNS: <dns> AWS Account: <AWS_id> AWS Billing Account: <AWS_id> API URL: <ocm_api> Console URL: Region: us-east-1 Availability: - Control Plane: MultiAZ - Data Plane: SingleAZ Nodes: - Compute (desired): 2 - Compute (current): 0 Network: - Type: OVNKubernetes - Service CIDR: <service_cidr> - Machine CIDR: <machine_cidr> - Pod CIDR: <pod_cidr> - Host Prefix: /23 - Subnets: <subnet_ids> EC2 Metadata Http Tokens: optional Role (STS) ARN: arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Installer-Role Support Role ARN: arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Support-Role Instance IAM Roles: - Worker: arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Worker-Role Operator IAM Roles: - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-cloud-network-config-controller-clo - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-capa-controller-manager - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-control-plane-operator - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-kms-provider - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-kube-controller-manager - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-image-registry-installer-cloud-cred - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-cluster-csi-drivers-ebs-cloud-crede Managed Policies: Yes State: ready Private: No Created: Mar 29 2024 14:25:52 UTC User Workload Monitoring: Enabled Details Page: https://<url> OIDC Endpoint URL: https://<endpoint> (Managed) Audit Log Forwarding: Disabled External Authentication: Enabled 1 rosa create external-auth-provider -c <cluster_name> I: Enabling interactive mode ? Name: 1 ? Issuer audiences: 2 ? The serving url of the token issuer: 3 ? CA file path (optional): 4 ? Claim mapping username: 5 ? Claim mapping groups: 6 ? Claim validation rule (optional): 7 ? Console client id (optional): 8 rosa create external-auth-provider --cluster=<cluster_id> --name=<provider_name> --issuer-url=<issuing_url> --issuer-audiences=<audience_id> --claim-mapping-username-claim=email --claim-mapping-groups-claim=groups --console-client-id=<client_id_for_app_registration> --console-client-secret=<client_secret> I: Successfully created an external authentication provider for cluster '<cluster_id>' rosa list external-auth-provider -c <cluster_name> NAME ISSUER URL m-entra-id https://login.microsoftonline.com/<group_id>/v2.0 rosa describe external-auth-provider -c <cluster_name> --name <name_of_external_authentication> ID: ms-entra-id Cluster ID: <cluster_id> Issuer audiences: - <audience_id> Issuer Url: https://login.microsoftonline.com/<group_id>/v2.0 Claim mappings group: groups Claim mappings username: email rosa create break-glass-credential -c <cluster_name> -i 1 I: Enabling interactive mode ? Username (optional): 1 ? Expiration duration (optional): 2 I: Successfully created a break glass credential for cluster 'ac-hcp-test'. rosa create break-glass-credential -c mycluster --username test-username --expiration 1h rosa list break-glass-credential -c mycluster ID USERNAME STATUS 2a7jli9n4phe6c02ul7ti91djtv2o51d test-user issued rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name> ID: 2a7jli9n4phe6c02ul7ti91djtv2o51d Username: test-user Expire at: Dec 28 2026 10:23:05 EDT Status: issued mkdir ~/kubeconfigs export CLUSTER_NAME=<cluster_name> && export KUBECONFIG=~/kubeconfigs/break-glass-USD{CLUSTER_NAME}.kubeconfig rosa describe break-glass-credential <break_glass_credential_id> -c mycluster --kubeconfig apiVersion: v1 clusters: - cluster: server: <server_url> name: cluster contexts: - context: cluster: cluster namespace: default user: test-username name: admin current-context: admin kind: Config preferences: {} users: - name: test-user user: client-certificate-data: <client-certificate-data> 1 client-key-data: <client-key-data> 2 rosa describe break-glass-credential <break_glass_credential_id> -c mycluster --kubeconfig > USDKUBECONFIG rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name> --kubeconfig > USDKUBECONFIG oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-0-27.ec2.internal Ready worker 8m v1.28.7+f1b5f6c ip-10-0-0-67.ec2.internal Ready worker 9m v1.28.7+f1b5f6c kubectl auth whoami ATTRIBUTE VALUE Username system:customer-break-glass:test-user Groups [system:masters system:authenticated] oc apply -f - <<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: rosa-hcp-admins roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: f715c264-ab90-45d5-8a29-2e91a609a895 EOF clusterrolebinding.rbac.authorization.k8s.io/rosa-hcp-admins created rosa revoke break-glass-credentials -c <cluster_name> 1 ? Are you sure you want to revoke all the break glass credentials on cluster 'my-cluster'?: Yes I: Successfully requested revocation for all break glass credentials from cluster 'my-cluster' rosa list break-glass-credential -c <cluster_name> ID USERNAME STATUS 2330dbs0n8m3chkkr25gkkcd8pnj3lk2 test-user awaiting_revocation rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name> ID: 2330dbs0n8m3chkkr25gkkcd8pnj3lk2 Username: test-user Expire at: Dec 28 2026 10:23:05 EDT Status: issued Revoked at: Dec 27 2026 15:30:33 EDT rosa list external-auth-provider -c <cluster_name> NAME ISSUER URL entra-test https://login.microsoftonline.com/<group_id>/v2.0 rosa delete external-auth-provider <name_of_provider> -c <cluster_name> ? Are you sure you want to delete external authentication provider entra-test on cluster rosa-ext-test? Yes I: Successfully deleted external authentication provider 'entra-test' from cluster 'rosa-ext-test' rosa list external-auth-provider -c <cluster_name> E: there are no external authentication providers for this cluster rosa create account-roles --hosted-cp export ACCOUNT_ROLES_PREFIX=<account_role_prefix> echo USDACCOUNT_ROLES_PREFIX ManagedOpenShift rosa create oidc-config --mode=auto --yes ? Would you like to create a Managed (Red Hat hosted) OIDC Configuration Yes I: Setting up managed OIDC configuration I: To create Operator Roles for this OIDC Configuration, run the following command and remember to replace <user-defined> with a prefix of your choice: rosa create operator-roles --prefix <user-defined> --oidc-config-id 13cdr6b If you are going to create a Hosted Control Plane cluster please include '--hosted-cp' I: Creating OIDC provider using 'arn:aws:iam::4540112244:user/userName' ? Create the OIDC provider? Yes I: Created OIDC provider with ARN 'arn:aws:iam::4540112244:oidc-provider/dvbwgdztaeq9o.cloudfront.net/13cdr6b' export OIDC_ID=<oidc_config_id> 1 echo USDOIDC_ID 13cdr6b rosa list oidc-config ID MANAGED ISSUER URL SECRET ARN 2330dbs0n8m3chkkr25gkkcd8pnj3lk2 true https://dvbwgdztaeq9o.cloudfront.net/2330dbs0n8m3chkkr25gkkcd8pnj3lk2 233hvnrjoqu14jltk6lhbhf2tj11f8un false https://oidc-r7u1.s3.us-east-1.amazonaws.com aws:secretsmanager:us-east-1:242819244:secret:rosa-private-key-oidc-r7u1-tM3MDN export OPERATOR_ROLES_PREFIX=<prefix_name> rosa create operator-roles --hosted-cp --prefix=USDOPERATOR_ROLES_PREFIX --oidc-config-id=USDOIDC_ID --installer-role-arn arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/USD{ACCOUNT_ROLES_PREFIX}-HCP-ROSA-Installer-Role rosa create operator-roles --hosted-cp --prefix=USDOPERATOR_ROLES_PREFIX 1 --oidc-config-id=USDOIDC_ID 2 --installer-role-arn arn:aws:iam::USD{AWS_ACCOUNT_ID}:role/USD{ACCOUNT_ROLES_PREFIX}-HCP-ROSA-Installer-Role 3 ? Role creation mode: auto ? Operator roles prefix: <pre-filled_prefix> 1 ? OIDC Configuration ID: 23soa2bgvpek9kmes9s7os0a39i13qm4 | https://dvbwgdztaeq9o.cloudfront.net/23soa2bgvpek9kmes9s7os0a39i13qm4 2 ? Create hosted control plane operator roles: Yes W: More than one Installer role found ? Installer role ARN: arn:aws:iam::4540112244:role/<prefix>-HCP-ROSA-Installer-Role ? Permissions boundary ARN (optional): I: Reusable OIDC Configuration detected. Validating trusted relationships to operator roles: I: Creating roles using 'arn:aws:iam::4540112244:user/<userName>' I: Created role '<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials' I: Created role '<prefix>-openshift-cloud-network-config-controller-cloud-credenti' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-cloud-network-config-controller-cloud-credenti' I: Created role '<prefix>-kube-system-kube-controller-manager' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-kube-controller-manager' I: Created role '<prefix>-kube-system-capa-controller-manager' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-capa-controller-manager' I: Created role '<prefix>-kube-system-control-plane-operator' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-control-plane-operator' I: Created role '<prefix>-kube-system-kms-provider' with ARN 'arn:aws:iam::4540112244:role/<prefix>-kube-system-kms-provider' I: Created role '<prefix>-openshift-image-registry-installer-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-image-registry-installer-cloud-credentials' I: Created role '<prefix>-openshift-ingress-operator-cloud-credentials' with ARN 'arn:aws:iam::4540112244:role/<prefix>-openshift-ingress-operator-cloud-credentials' I: To create a cluster with these roles, run the following command: rosa create cluster --sts --oidc-config-id 23soa2bgvpek9kmes9s7os0a39i13qm4 --operator-roles-prefix <prefix> --hosted-cp rosa list operator-roles I: Fetching operator roles ROLE PREFIX AMOUNT IN BUNDLE <prefix> 8 ? Would you like to detail a specific prefix Yes 1 ? Operator Role Prefix: <prefix> ROLE NAME ROLE ARN VERSION MANAGED <prefix>-kube-system-capa-controller-manager arn:aws:iam::4540112244:role/<prefix>-kube-system-capa-controller-manager 4.13 No <prefix>-kube-system-control-plane-operator arn:aws:iam::4540112244:role/<prefix>-kube-system-control-plane-operator 4.13 No <prefix>-kube-system-kms-provider arn:aws:iam::4540112244:role/<prefix>-kube-system-kms-provider 4.13 No <prefix>-kube-system-kube-controller-manager arn:aws:iam::4540112244:role/<prefix>-kube-system-kube-controller-manager 4.13 No <prefix>-openshift-cloud-network-config-controller-cloud-credenti arn:aws:iam::4540112244:role/<prefix>-openshift-cloud-network-config-controller-cloud-credenti 4.13 No <prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-cluster-csi-drivers-ebs-cloud-credentials 4.13 No <prefix>-openshift-image-registry-installer-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-image-registry-installer-cloud-credentials 4.13 No <prefix>-openshift-ingress-operator-cloud-credentials arn:aws:iam::4540112244:role/<prefix>-openshift-ingress-operator-cloud-credentials 4.13 No rosa create cluster --cluster-name=<cluster_name> --sts --mode=auto --hosted-cp --operator-roles-prefix <operator-role-prefix> --oidc-config-id <ID-of-OIDC-configuration> --subnet-ids=<public-subnet-id>,<private-subnet-id> --no-cni rosa create cluster --private --cluster-name=<cluster_name> --sts --mode=auto --hosted-cp --subnet-ids=<private-subnet-id> --no-cni rosa create cluster --hosted-cp --subnet-ids=USDSUBNET_IDS --oidc-config-id=USDOIDC_ID --cluster-name=<cluster_name> --operator-roles-prefix=USDOPERATOR_ROLES_PREFIX --no-cni rosa describe cluster --cluster=<cluster_name> rosa logs install --cluster=<cluster_name> --watch 1 rosa describe cluster --cluster=<cluster_name> Name: test_cluster Domain Prefix: test_cluster Display Name: test_cluster ID: <cluster_id> 1 External ID: <external_id> Control Plane: ROSA Service Hosted OpenShift Version: 4.18.0 Channel Group: stable DNS: test_cluster.l3cn.p3.openshiftapps.com AWS Account: <AWS_id> AWS Billing Account: <AWS_id> API URL: https://api.test_cluster.l3cn.p3.openshiftapps.com:443 Console URL: Region: us-east-1 Availability: - Control Plane: MultiAZ - Data Plane: SingleAZ Nodes: - Compute (desired): 2 - Compute (current): 0 Network: - Type: OVNKubernetes - Service CIDR: 172.30.0.0/16 - Machine CIDR: 10.0.0.0/16 - Pod CIDR: 10.128.0.0/14 - Host Prefix: /23 - Subnets: <subnet_ids> EC2 Metadata Http Tokens: optional Role (STS) ARN: arn:aws:iam::<AWS_id>:role/test_cluster-HCP-ROSA-Installer-Role Support Role ARN: arn:aws:iam::<AWS_id>:role/test_cluster-HCP-ROSA-Support-Role Instance IAM Roles: - Worker: arn:aws:iam::<AWS_id>:role/test_cluster-HCP-ROSA-Worker-Role Operator IAM Roles: 2 - arn:aws:iam::<AWS_id>:role/test_cluster-openshift-cloud-network-config-controller-cloud-crede - arn:aws:iam::<AWS_id>:role/test_cluster-openshift-image-registry-installer-cloud-credentials - arn:aws:iam::<AWS_id>:role/test_cluster-openshift-ingress-operator-cloud-credentials - arn:aws:iam::<AWS_id>:role/test_cluster-kube-system-kube-controller-manager - arn:aws:iam::<AWS_id>:role/test_cluster-kube-system-capa-controller-manager - arn:aws:iam::<AWS_id>:role/test_cluster-kube-system-control-plane-operator - arn:aws:iam::<AWS_id>:role/hcpcluster-kube-system-kms-provider - arn:aws:iam::<AWS_id>:role/test_cluster-openshift-cluster-csi-drivers-ebs-cloud-credentials Managed Policies: Yes State: ready Private: No Created: Apr 16 2024 20:32:06 UTC User Workload Monitoring: Enabled Details Page: https://console.redhat.com/openshift/details/s/<cluster_id> OIDC Endpoint URL: https://oidc.op1.openshiftapps.com/<cluster_id> (Managed) 3 Audit Log Forwarding: Disabled External Authentication: Disabled rosa delete cluster --cluster=<cluster_name> --watch rosa delete operator-roles --prefix <operator_role_prefix> rosa delete oidc-provider --oidc-config-id <oidc_config_id> rosa list account-roles I: Fetching account roles ROLE NAME ROLE TYPE ROLE ARN OPENSHIFT VERSION AWS Managed ManagedOpenShift-HCP-ROSA-Installer-Role Installer arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-HCP-ROSA-Installer-Role 4.18 Yes ManagedOpenShift-HCP-ROSA-Support-Role Support arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-HCP-ROSA-Support-Role 4.18 Yes ManagedOpenShift-HCP-ROSA-Worker-Role Worker arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-HCP-ROSA-Worker-Role 4.18 Yes rosa delete account-roles --prefix <prefix> --mode auto 1 W: There are no classic account roles to be deleted I: Deleting hosted CP account roles ? Delete the account role 'delete-rosa-HCP-ROSA-Installer-Role'? Yes I: Deleting account role 'delete-rosa-HCP-ROSA-Installer-Role' ? Delete the account role 'delete-rosa-HCP-ROSA-Support-Role'? Yes I: Deleting account role 'delete-rosa-HCP-ROSA-Support-Role' ? Delete the account role 'delete-rosa-HCP-ROSA-Worker-Role'? Yes I: Deleting account role 'delete-rosa-HCP-ROSA-Worker-Role' I: Successfully deleted the hosted CP account roles rosa list ocm-roles I: Fetching ocm roles ROLE NAME ROLE ARN LINKED ADMIN AWS Managed ManagedOpenShift-OCM-Role-<red_hat_organization_external_id> arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id> Yes Yes Yes rosa unlink ocm-role --role-arn <arn> 1 I: Unlinking OCM role ? Unlink the 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>' role from organization '<red_hat_organization_id>'? Yes I: Successfully unlinked role-arn 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>' from organization account '<red_hat_organization_id>' rosa delete ocm-role --role-arn <arn> I: Deleting OCM role ? OCM Role ARN: arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id> ? Delete 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-OCM-Role-<red_hat_organization_external_id>' ocm role? Yes ? OCM role deletion mode: auto 1 I: Successfully deleted the OCM role rosa list user-roles I: Fetching user roles ROLE NAME ROLE ARN LINKED ManagedOpenShift-User-<ocm_user_name>-Role arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<ocm_user_name>-Role Yes rosa unlink user-role --role-arn <arn> 1 I: Unlinking user role ? Unlink the 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<ocm_user_name>-Role' role from the current account '<ocm_user_account_id>'? Yes I: Successfully unlinked role ARN 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<ocm_user_name>-Role' from account '<ocm_user_account_id>' rosa delete user-role --role-arn <arn> I: Deleting user role ? User Role ARN: arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<ocm_user_name>-Role ? Delete the 'arn:aws:iam::<aws_account_id>:role/ManagedOpenShift-User-<ocm_user_name>-Role' role from the AWS account? Yes ? User role deletion mode: auto 1 I: Successfully deleted the user role