Chapter 10. Management of Ceph object gateway using the Ceph Orchestrator
Chapter 10. Management of Ceph object gateway using the Ceph Orchestrator As a storage administrator, you can deploy Ceph object gateway using the command line interface or by using the service specification. You can also configure multi-site object gateways, and remove the Ceph object gateway using the Ceph Orchestrator. Cephadm deploys Ceph object gateway as a collection of daemons that manages a single-cluster deployment or a particular realm and zone in a multi-site deployment. Note With Cephadm, the object gateway daemons are configured using the monitor configuration database instead of a ceph.conf or the command line. If that configuration is not already in the client.rgw section, then the object gateway daemons will start up with default settings and bind to the port 80 . Note The .default.rgw.buckets.index pool is created only after the bucket is created in Ceph Object Gateway, while the .default.rgw.buckets.data pool is created after the data is uploaded to the bucket. This section covers the following administrative tasks: Deploying the Ceph object gateway using the command line interface . Deploying the Ceph object gateway using the service specification . Deploying a multi-site Ceph object gateway using the Ceph Orchestrator . Removing the Ceph object gateway using the Ceph Orchestrator . Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to all the nodes. Hosts are added to the cluster. All the managers, monitors, and OSDs are deployed in the storage cluster. 10.1. Deploying the Ceph Object Gateway using the command line interface Using the Ceph Orchestrator, you can deploy the Ceph Object Gateway with the ceph orch command in the command line interface. Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to all the nodes. Hosts are added to the cluster. All manager, monitor and OSD daemons are deployed. Procedure Log into the Cephadm shell: Example You can deploy the Ceph object gateway daemons in three different ways: Method 1 Create realm, zone group, zone, and then use the placement specification with the host name: Create a realm: Syntax Example Create a zone group: Syntax Example Create a zone: Syntax Example Commit the changes: Syntax Example Run the ceph orch apply command: Syntax Example Method 2 Use an arbitrary service name to deploy two Ceph Object Gateway daemons for a single cluster deployment: Syntax Example Method 3 Use an arbitrary service name on a labeled set of hosts: Syntax Note NUMBER_OF_DAEMONS controls the number of Ceph object gateways deployed on each host. To achieve the highest performance without incurring an additional cost, set this value to 2. Example Verification List the service: Example List the hosts, daemons, and processes: Syntax Example 10.2. Deploying the Ceph Object Gateway using the service specification You can deploy the Ceph Object Gateway using the service specification with either the default or the custom realms, zones, and zone groups. Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to the bootstrapped host. Hosts are added to the cluster. All manager, monitor, and OSD daemons are deployed. Procedure As a root user, create a specification file: Example Edit the radosgw.yml file to include the following details for the default realm, zone, and zone group: Syntax Note NUMBER_OF_DAEMONS controls the number of Ceph Object Gateways deployed on each host. To achieve the highest performance without incurring an additional cost, set this value to 2. Example Optional: For custom realm, zone, and zone group, create the resources and then create the radosgw.yml file: Create the custom realm, zone, and zone group: Example Create the radosgw.yml file with the following details: Example Mount the radosgw.yml file under a directory in the container: Example Note Every time you exit the shell, you have to mount the file in the container before deploying the daemon. Deploy the Ceph Object Gateway using the service specification: Syntax Example Verification List the service: Example List the hosts, daemons, and processes: Syntax Example 10.3. Deploying a multi-site Ceph Object Gateway using the Ceph Orchestrator Ceph Orchestrator supports multi-site configuration options for the Ceph Object Gateway. You can configure each object gateway to work in an active-active zone configuration allowing writes to a non-primary zone. The multi-site configuration is stored within a container called a realm. The realm stores zone groups, zones, and a time period. The rgw daemons handle the synchronization eliminating the need for a separate synchronization agent, thereby operating with an active-active configuration. You can also deploy multi-site zones using the command line interface (CLI). Note The following configuration assumes at least two Red Hat Ceph Storage clusters are in geographically separate locations. However, the configuration also works on the same site. Prerequisites At least two running Red Hat Ceph Storage clusters. At least two Ceph Object Gateway instances, one for each Red Hat Ceph Storage cluster. Root-level access to all the nodes. Nodes or containers are added to the storage cluster. All Ceph Manager, Monitor and OSD daemons are deployed. Procedure In the cephadm shell, configure the primary zone: Create a realm: Syntax Example If the storage cluster has a single realm, then specify the --default flag. Create a primary zone group: Syntax Example Create a primary zone: Syntax Example Optional: Delete the default zone, zone group, and the associated pools. Important Do not delete the default zone and its pools if you are using the default zone and zone group to store data. Also, removing the default zone group deletes the system user. To access old data in the default zone and zonegroup, use --rgw-zone default and --rgw-zonegroup default in radosgw-admin commands. Example Create a system user: Syntax Example Make a note of the access_key and secret_key . Add the access key and system key to the primary zone: Syntax Example Commit the changes: Syntax Example Outside the cephadm shell, fetch the FSID of the storage cluster and the processes: Example Start the Ceph Object Gateway daemon: Syntax Example In the Cephadm shell, configure the secondary zone. Pull the primary realm configuration from the host: Syntax Example Pull the primary period configuration from the host: Syntax Example Configure a secondary zone: Syntax Example Optional: Delete the default zone. Important Do not delete the default zone and its pools if you are using the default zone and zone group to store data. To access old data in the default zone and zonegroup, use --rgw-zone default and --rgw-zonegroup default in radosgw-admin commands. Example Update the Ceph configuration database: Syntax Example Commit the changes: Syntax Example Outside the Cephadm shell, fetch the FSID of the storage cluster and the processes: Example Start the Ceph Object Gateway daemon: Syntax Example Optional: Deploy multi-site Ceph Object Gateways using the placement specification: Syntax Example Verification Check the synchronization status to verify the deployment: Example 10.4. Removing the Ceph Object Gateway using the Ceph Orchestrator You can remove the Ceph object gateway daemons using the ceph orch rm command. Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to all the nodes. Hosts are added to the cluster. At least one Ceph object gateway daemon deployed on the hosts. Procedure Log into the Cephadm shell: Example List the service: Example Remove the service: Syntax Example Verification List the hosts, daemons, and processes: Syntax Example Additional Resources See Deploying the Ceph object gateway using the command line interface section in the Red Hat Ceph Storage Operations Guide for more information. See Deploying the Ceph object gateway using the service specification section in the Red Hat Ceph Storage Operations Guide for more information.
cephadm shell radosgw-admin realm create --rgw-realm= REALM_NAME --default radosgw-admin realm create --rgw-realm=test_realm --default radosgw-admin zonegroup create --rgw-zonegroup= ZONE_GROUP_NAME --master --default radosgw-admin zonegroup create --rgw-zonegroup=default --master --default radosgw-admin zone create --rgw-zonegroup= ZONE_GROUP_NAME --rgw-zone= ZONE_NAME --master --default radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=test_zone --master --default radosgw-admin period update --rgw-realm= REALM_NAME --commit radosgw-admin period update --rgw-realm=test_realm --commit ceph orch apply rgw NAME [--realm= REALM_NAME ] [--zone= ZONE_NAME ] --placement=" NUMBER_OF_DAEMONS [ HOST_NAME_1 HOST_NAME_2 ]" ceph orch apply rgw test --realm=test_realm --zone=test_zone --placement="2 host01 host02" ceph orch apply rgw SERVICE_NAME ceph orch apply rgw foo ceph orch host label add HOST_NAME_1 LABEL_NAME ceph orch host label add HOSTNAME_2 LABEL_NAME ceph orch apply rgw SERVICE_NAME --placement="label: LABEL_NAME count-per-host: NUMBER_OF_DAEMONS " --port=8000 ceph orch host label add host01 rgw # the 'rgw' label can be anything ceph orch host label add host02 rgw ceph orch apply rgw foo --placement="2 label:rgw" --port=8000 ceph orch ls ceph orch ps --daemon_type= DAEMON_NAME ceph orch ps --daemon_type=rgw touch radosgw.yml service_type: rgw service_id: REALM_NAME . ZONE_NAME placement: hosts: - HOST_NAME_1 - HOST_NAME_2 count_per_host: NUMBER_OF_DAEMONS spec: rgw_realm: REALM_NAME rgw_zone: ZONE_NAME rgw_frontend_port: FRONT_END_PORT networks: - NETWORK_CIDR # Ceph Object Gateway service binds to a specific network service_type: rgw service_id: default placement: hosts: - host01 - host02 - host03 count_per_host: 2 spec: rgw_realm: default rgw_zone: default rgw_frontend_port: 1234 networks: - 192.169.142.0/24 radosgw-admin realm create --rgw-realm=test_realm radosgw-admin zonegroup create --rgw-zonegroup=test_zonegroup radosgw-admin zone create --rgw-zonegroup=test_zonegroup --rgw-zone=test_zone radosgw-admin period update --rgw-realm=test_realm --commit service_type: rgw service_id: test_realm.test_zone placement: hosts: - host01 - host02 - host03 count_per_host: 2 spec: rgw_realm: test_realm rgw_zone: test_zone rgw_frontend_port: 1234 networks: - 192.169.142.0/24 cephadm shell --mount radosgw.yml:/var/lib/ceph/radosgw/radosgw.yml ceph orch apply -i FILE_NAME .yml ceph orch apply -i radosgw.yml ceph orch ls ceph orch ps --daemon_type= DAEMON_NAME ceph orch ps --daemon_type=rgw radosgw-admin realm create --rgw-realm= REALM_NAME --default radosgw-admin realm create --rgw-realm=test_realm --default radosgw-admin zonegroup create --rgw-zonegroup= ZONE_GROUP_NAME --endpoints=http:// RGW_PRIMARY_HOSTNAME : RGW_PRIMARY_PORT_NUMBER_1 --master --default radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --master --default radosgw-admin zone create --rgw-zonegroup= PRIMARY_ZONE_GROUP_NAME --rgw-zone= PRIMARY_ZONE_NAME --endpoints=http:// RGW_PRIMARY_HOSTNAME : RGW_PRIMARY_PORT_NUMBER_1 --access-key= SYSTEM_ACCESS_KEY --secret= SYSTEM_SECRET_KEY radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-1 --endpoints=http://rgw1:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ radosgw-admin zonegroup delete --rgw-zonegroup=default ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it radosgw-admin user create --uid= USER_NAME --display-name=" USER_NAME " --access-key= SYSTEM_ACCESS_KEY --secret= SYSTEM_SECRET_KEY --system radosgw-admin user create --uid=zone.user --display-name="Zone user" --system radosgw-admin zone modify --rgw-zone= PRIMARY_ZONE_NAME --access-key= ACCESS_KEY --secret= SECRET_KEY radosgw-admin zone modify --rgw-zone=us-east-1 --access-key=NE48APYCAODEPLKBCZVQ--secret=u24GHQWRE3yxxNBnFBzjM4jn14mFIckQ4EKL6LoW radosgw-admin period update --commit radosgw-admin period update --commit systemctl list-units | grep ceph systemctl start ceph- FSID @ DAEMON_NAME systemctl enable ceph- FSID @ DAEMON_NAME systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service radosgw-admin realm pull --url= URL_TO_PRIMARY_ZONE_GATEWAY --access-key= ACCESS_KEY --secret-key= SECRET_KEY radosgw-admin realm pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ radosgw-admin period pull --url= URL_TO_PRIMARY_ZONE_GATEWAY --access-key= ACCESS_KEY --secret-key= SECRET_KEY radosgw-admin period pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ radosgw-admin zone create --rgw-zonegroup= ZONE_GROUP_NAME --rgw-zone= SECONDARY_ZONE_NAME --endpoints=http:// RGW_SECONDARY_HOSTNAME : RGW_PRIMARY_PORT_NUMBER_1 --access-key= SYSTEM_ACCESS_KEY --secret= SYSTEM_SECRET_KEY --endpoints=http:// FQDN :80 [--read-only] radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-2 --endpoints=http://rgw2:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ --endpoints=http://rgw.example.com:80 radosgw-admin zone rm --rgw-zone=default ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it ceph config set SERVICE_NAME rgw_zone SECONDARY_ZONE_NAME ceph config set rgw rgw_zone us-east-2 radosgw-admin period update --commit radosgw-admin period update --commit systemctl list-units | grep ceph systemctl start ceph- FSID @ DAEMON_NAME systemctl enable ceph- FSID @ DAEMON_NAME systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service ceph orch apply rgw NAME --realm= REALM_NAME --zone= PRIMARY_ZONE_NAME --placement=" NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2 " ceph orch apply rgw east --realm=test_realm --zone=us-east-1 --placement="2 host01 host02" radosgw-admin sync status cephadm shell ceph orch ls ceph orch rm SERVICE_NAME ceph orch rm rgw.test_realm.test_zone_bb ceph orch ps ceph orch ps