Chapter 4. Ceph Object Gateway and the Swift API
Chapter 4. Ceph Object Gateway and the Swift API As a developer, you can use a RESTful application programming interface (API) that is compatible with the Swift API data access model. You can manage the buckets and objects stored in Red Hat Ceph Storage cluster through the Ceph Object Gateway. The following table describes the support status for current Swift functional features: Table 4.1. Features Feature Status Remarks Authentication Supported Get Account Metadata Supported No custom metadata Swift ACLs Supported Supports a subset of Swift ACLs List Containers Supported List Container's Objects Supported Create Container Supported Delete Container Supported Get Container Metadata Supported Add/Update Container Metadata Supported Delete Container Metadata Supported Get Object Supported Create/Update an Object Supported Create Large Object Supported Delete Object Supported Copy Object Supported Get Object Metadata Supported Add/Update Object Metadata Supported Temp URL Operations Supported CORS Not Supported Expiring Objects Supported Object Versioning Not Supported Static Website Not Supported 4.1. Prerequisites A running Red Hat Ceph Storage cluster. A RESTful client. 4.2. Swift API limitations Important The following limitations should be used with caution. There are implications related to your hardware selections, so you should always discuss these requirements with your Red Hat account team. Maximum object size when using Swift API: 5GB Maximum metadata size when using Swift API: There is no defined limit on the total size of user metadata that can be applied to an object, but a single HTTP request is limited to 16,000 bytes. 4.3. Create a Swift user To test the Swift interface, create a Swift subuser. Creating a Swift user is a two-step process. The first step is to create the user. The second step is to create the secret key. Note In a multi-site deployment, always create a user on a host in the master zone of the master zone group. Prerequisites Installation of the Ceph Object Gateway. Root-level access to the Ceph Object Gateway node. Procedure Create the Swift user: Syntax Replace NAME with the Swift user name, for example: Example Create the secret key: Syntax Replace NAME with the Swift user name, for example: Example 4.4. Swift authenticating a user To authenticate a user, make a request containing an X-Auth-User and a X-Auth-Key in the header. Syntax Example Response Note You can retrieve data about Ceph's Swift-compatible service by executing GET requests using the X-Storage-Url value during authentication. Additional Resources See the Red Hat Ceph Storage Developer Guide for Swift request headers. See the Red Hat Ceph Storage Developer Guide for Swift response headers. 4.5. Swift container operations As a developer, you can perform container operations with the Swift application programming interface (API) through the Ceph Object Gateway. You can list, create, update, and delete containers. You can also add or update the container's metadata. 4.5.1. Prerequisites A running Red Hat Ceph Storage cluster. A RESTful client. 4.5.2. Swift container operations A container is a mechanism for storing data objects. An account can have many containers, but container names must be unique. This API enables a client to create a container, set access controls and metadata, retrieve a container's contents, and delete a container. Since this API makes requests related to information in a particular user's account, all requests in this API must be authenticated unless a container's access control is deliberately made publicly accessible, that is, allows anonymous requests. Note The Amazon S3 API uses the term 'bucket' to describe a data container. When you hear someone refer to a 'bucket' within the Swift API, the term 'bucket' might be construed as the equivalent of the term 'container.' One facet of object storage is that it does not support hierarchical paths or directories. Instead, it supports one level consisting of one or more containers, where each container might have objects. The RADOS Gateway's Swift-compatible API supports the notion of 'pseudo-hierarchical containers', which is a means of using object naming to emulate a container, or directory hierarchy without actually implementing one in the storage system. You can name objects with pseudo-hierarchical names, for example, photos/buildings/empire-state.jpg, but container names cannot contain a forward slash ( / ) character. Important When uploading large objects to versioned Swift containers, use the --leave-segments option with the python-swiftclient utility. Not using --leave-segments overwrites the manifest file. Consequently, an existing object is overwritten, which leads to data loss. 4.5.3. Swift update a container's Access Control List (ACL) When a user creates a container, the user has read and write access to the container by default. To allow other users to read a container's contents or write to a container, you must specifically enable the user. You can also specify * in the X-Container-Read or X-Container-Write settings, which effectively enables all users to either read from or write to the container. Setting * makes the container public. That is it enables anonymous users to either read from or write to the container. Syntax Request Headers X-Container-Read Description The user IDs with read permissions for the container. Type Comma-separated string values of user IDs. Required No X-Container-Write Description The user IDs with write permissions for the container. Type Comma-separated string values of user IDs. Required No 4.5.4. Swift list containers A GET request that specifies the API version and the account will return a list of containers for a particular user account. Since the request returns a particular user's containers, the request requires an authentication token. The request cannot be made anonymously. Syntax Request Parameters limit Description Limits the number of results to the specified value. Type Integer Valid Values N/A Required Yes format Description Limits the number of results to the specified value. Type Integer Valid Values json or xml Required No marker Description Returns a list of results greater than the marker value. Type String Valid Values N/A Required No The response contains a list of containers, or returns with an HTTP 204 response code. Response Entities account Description A list for account information. Type Container container Description The list of containers. Type Container name Description The name of a container. Type String bytes Description The size of the container. Type Integer 4.5.5. Swift list a container's objects To list the objects within a container, make a GET request with the API version, account, and the name of the container. You can specify query parameters to filter the full list, or leave out the parameters to return a list of the first 10,000 object names stored in the container. Syntax Request Parameters format Description Limits the number of results to the specified value. Type Integer Valid Values json or xml Required No prefix Description Limits the result set to objects beginning with the specified prefix. Type String Valid Values N/A Required No marker Description Returns a list of results greater than the marker value. Type String Valid Values N/A Required No limit Description Limits the number of results to the specified value. Type Integer Valid Values 0 - 10,000 Required No delimiter Description The delimiter between the prefix and the rest of the object name. Type String Valid Values N/A Required No path Description The pseudo-hierarchical path of the objects. Type String Valid Values N/A Required No Response Entities container Description The container. Type Container object Description An object within the container. Type Container name Description The name of an object within the container. Type String hash Description A hash code of the object's contents. Type String last_modified Description The last time the object's contents were modified. Type Date content_type Description The type of content within the object. Type String 4.5.6. Swift create a container To create a new container, make a PUT request with the API version, account, and the name of the new container. The container name must be unique, must not contain a forward-slash (/) character, and should be less than 256 bytes. You can include access control headers and metadata headers in the request. You can also include a storage policy identifying a key for a set of placement pools. For example, execute radosgw-admin zone get to see a list of available keys under placement_pools . A storage policy enables you to specify a special set of pools for the container, for example, SSD-based storage. The operation is idempotent. If you make a request to create a container that already exists, it will return with a HTTP 202 return code, but will not create another container. Syntax Headers X-Container-Read Description The user IDs with read permissions for the container. Type Comma-separated string values of user IDs. Required No X-Container-Write Description The user IDs with write permissions for the container. Type Comma-separated string values of user IDs. Required No X-Container-Meta- KEY Description A user-defined metadata key that takes an arbitrary string value. Type String Required No X-Storage-Policy Description The key that identifies the storage policy under placement_pools for the Ceph Object Gateway. Execute radosgw-admin zone get for available keys. Type String Required No If a container with the same name already exists, and the user is the container owner then the operation will succeed. Otherwise, the operation will fail. HTTP Response 409 Status Code BucketAlreadyExists Description The container already exists under a different user's ownership. 4.5.7. Swift delete a container To delete a container, make a DELETE request with the API version, account, and the name of the container. The container must be empty. If you'd like to check if the container is empty, execute a HEAD request against the container. Once you've successfully removed the container, you'll be able to reuse the container name. Syntax HTTP Response 204 Status Code NoContent Description The container was removed. 4.5.8. Swift add or update the container metadata To add metadata to a container, make a POST request with the API version, account, and container name. You must have write permissions on the container to add or update metadata. Syntax Request Headers X-Container-Meta- KEY Description A user-defined metadata key that takes an arbitrary string value. Type String Required No 4.6. Swift object operations As a developer, you can perform object operations with the Swift application programming interface (API) through the Ceph Object Gateway. You can list, create, update, and delete objects. You can also add or update the object's metadata. 4.6.1. Prerequisites A running Red Hat Ceph Storage cluster. A RESTful client. 4.6.2. Swift object operations An object is a container for storing data and metadata. A container might have many objects, but the object names must be unique. This API enables a client to create an object, set access controls and metadata, retrieve an object's data and metadata, and delete an object. Since this API makes requests related to information in a particular user's account, all requests in this API must be authenticated. Unless the container or object's access control is deliberately made publicly accessible, that is, allows anonymous requests. 4.6.3. Swift get an object To retrieve an object, make a GET request with the API version, account, container, and object name. You must have read permissions on the container to retrieve an object within it. Syntax Request Headers range Description To retrieve a subset of an object's contents, you can specify a byte range. Type Date Required No If-Modified-Since Description Only copies if modified since the date and time of the source object's last_modified attribute. Type Date Required No If-Unmodified-Since Description Only copies if not modified since the date and time of the source object's last_modified attribute. Type Date Required No Copy-If-Match Description Copies only if the ETag in the request matches the source object's ETag. Type ETag Required No Copy-If-None-Match Description Copies only if the ETag in the request does not match the source object's ETag. Type ETag Required No Response Headers Content-Range Description The range of the subset of object contents. Returned only if the range header field was specified in the request. 4.6.4. Swift create or update an object To create a new object, make a PUT request with the API version, account, container name, and the name of the new object. You must have write permission on the container to create or update an object. The object name must be unique within the container. The PUT request is not idempotent, so if you do not use a unique name, the request will update the object. However, you can use pseudo-hierarchical syntax in the object name to distinguish it from another object of the same name if it is under a different pseudo-hierarchical directory. You can include access control headers and metadata headers in the request. Syntax Request Headers ETag Description An MD5 hash of the object's contents. Recommended. Type String Valid Values N/A Required No Content-Type Description An MD5 hash of the object's contents. Type String Valid Values N/A Required No Transfer-Encoding Description Indicates whether the object is part of a larger aggregate object. Type String Valid Values chunked Required No 4.6.5. Swift delete an object To delete an object, make a DELETE request with the API version, account, container, and object name. You must have write permissions on the container to delete an object within it. Once you've successfully deleted the object, you will be able to reuse the object name. Syntax 4.6.6. Swift copy an object Copying an object allows you to make a server-side copy of an object, so that you do not have to download it and upload it under another container. To copy the contents of one object to another object, you can make either a PUT request or a COPY request with the API version, account, and the container name. For a PUT request, use the destination container and object name in the request, and the source container and object in the request header. For a Copy request, use the source container and object in the request, and the destination container and object in the request header. You must have write permission on the container to copy an object. The destination object name must be unique within the container. The request is not idempotent, so if you do not use a unique name, the request will update the destination object. You can use pseudo-hierarchical syntax in the object name to distinguish the destination object from the source object of the same name if it is under a different pseudo-hierarchical directory. You can include access control headers and metadata headers in the request. Syntax or alternatively: Syntax Request Headers X-Copy-From Description Used with a PUT request to define the source container/object path. Type String Required Yes, if using PUT . Destination Description Used with a COPY request to define the destination container/object path. Type String Required Yes, if using COPY . If-Modified-Since Description Only copies if modified since the date and time of the source object's last_modified attribute. Type Date Required No If-Unmodified-Since Description Only copies if not modified since the date and time of the source object's last_modified attribute. Type Date Required No Copy-If-Match Description Copies only if the ETag in the request matches the source object's ETag. Type ETag Required No Copy-If-None-Match Description Copies only if the ETag in the request does not match the source object's ETag. Type ETag Required No 4.6.7. Swift get object metadata To retrieve an object's metadata, make a HEAD request with the API version, account, container, and object name. You must have read permissions on the container to retrieve metadata from an object within the container. This request returns the same header information as the request for the object itself, but it does not return the object's data. Syntax 4.6.8. Swift add or update object metadata To add metadata to an object, make a POST request with the API version, account, container, and object name. You must have write permissions on the parent container to add or update metadata. Syntax Request Headers X-Object-Meta- KEY Description A user-defined meta data key that takes an arbitrary string value. Type String Required No 4.7. Swift temporary URL operations To allow temporary access, temp url functionality is supported by swift endpoint of radosgw . For example GET requests, to objects without the need to share credentials. For this functionality, initially the value of X-Account-Meta-Temp-URL-Key and optionally X-Account-Meta-Temp-URL-Key-2 should be set. The Temp URL functionality relies on a HMAC-SHA1 signature against these secret keys. 4.7.1. Swift get temporary URL objects Temporary URL uses a cryptographic HMAC-SHA1 signature, which includes the following elements: The value of the Request method, "GET" for instance The expiry time, in the format of seconds since the epoch, that is, Unix time The request path starting from "v1" onwards The above items are normalized with newlines appended between them, and a HMAC is generated using the SHA-1 hashing algorithm against one of the Temp URL Keys posted earlier. A sample python script to demonstrate the above is given below: Example Example Output 4.7.2. Swift POST temporary URL keys A POST request to the swift account with the required Key will set the secret temp URL key for the account against which temporary URL access can be provided to accounts. Up to two keys are supported, and signatures are checked against both the keys, if present, so that keys can be rotated without invalidating the temporary URLs. Syntax Request Headers X-Account-Meta-Temp-URL-Key Description A user-defined key that takes an arbitrary string value. Type String Required Yes X-Account-Meta-Temp-URL-Key-2 Description A user-defined key that takes an arbitrary string value. Type String Required No 4.8. Swift multi-tenancy container operations When a client application accesses containers, it always operates with credentials of a particular user. In Red Hat Ceph Storage cluster, every user belongs to a tenant. Consequently, every container operation has an implicit tenant in its context if no tenant is specified explicitly. Thus multi-tenancy is completely backward compatible with releases, as long as the referred containers and referring user belong to the same tenant. Extensions employed to specify an explicit tenant differ according to the protocol and authentication system used. A colon character separates tenant and container, thus a sample URL would be: Example By contrast, in a create_container() method, simply separate the tenant and container in the container method itself: Example 4.9. Additional Resources See the Red Hat Ceph Storage Object Gateway Guide for details on multi-tenancy. See Appendix E, Swift request headers for Swift request headers. See Appendix F, Swift response headers for Swift response headers.
radosgw-admin subuser create --uid= NAME --subuser= NAME :swift --access=full radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full { "user_id": "testuser", "display_name": "First User", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [ { "id": "testuser:swift", "permissions": "full-control" } ], "keys": [ { "user": "testuser", "access_key": "O8JDE41XMI74O185EHKD", "secret_key": "i4Au2yxG5wtr1JK01mI8kjJPM93HNAoVWOSTdJd6" } ], "swift_keys": [ { "user": "testuser:swift", "secret_key": "13TLtdEW7bCqgttQgPzxFxziu0AgabtOc6vM8DLA" } ], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } radosgw-admin key create --subuser= NAME :swift --key-type=swift --gen-secret radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret { "user_id": "testuser", "display_name": "First User", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [ { "id": "testuser:swift", "permissions": "full-control" } ], "keys": [ { "user": "testuser", "access_key": "O8JDE41XMI74O185EHKD", "secret_key": "i4Au2yxG5wtr1JK01mI8kjJPM93HNAoVWOSTdJd6" } ], "swift_keys": [ { "user": "testuser:swift", "secret_key": "a4ioT4jEP653CDcdU8p4OuhruwABBRZmyNUbnSSt" } ], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } GET /auth HTTP/1.1 Host: swift.example.com X-Auth-User: johndoe X-Auth-Key: R7UUOLFDI2ZI9PRCQ53K HTTP/1.1 204 No Content Date: Mon, 16 Jul 2012 11:05:33 GMT Server: swift X-Storage-Url: https://swift.example.com X-Storage-Token: UOlCCC8TahFKlWuv9DB09TWHF0nDjpPElha0kAa Content-Length: 0 Content-Type: text/plain; charset=UTF-8 POST / API_VERSION / ACCOUNT / TENANT : CONTAINER HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN X-Container-Read: * X-Container-Write: UID1 , UID2 , UID3 GET / API_VERSION / ACCOUNT HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN GET / API_VERSION / TENANT : CONTAINER HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN PUT / API_VERSION / ACCOUNT / TENANT : CONTAINER HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN X-Container-Read: COMMA_SEPARATED_UIDS X-Container-Write: COMMA_SEPARATED_UIDS X-Container-Meta- KEY : VALUE X-Storage-Policy: PLACEMENT_POOLS_KEY DELETE / API_VERSION / ACCOUNT / TENANT : CONTAINER HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN POST / API_VERSION / ACCOUNT / TENANT : CONTAINER HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN X-Container-Meta-Color: red X-Container-Meta-Taste: salty GET / API_VERSION / ACCOUNT / TENANT : CONTAINER / OBJECT HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN PUT / API_VERSION / ACCOUNT / TENANT : CONTAINER HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN DELETE / API_VERSION / ACCOUNT / TENANT : CONTAINER / OBJECT HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN PUT / API_VERSION / ACCOUNT / TENANT : CONTAINER HTTP/1.1 X-Copy-From: TENANT : SOURCE_CONTAINER / SOURCE_OBJECT Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN COPY / API_VERSION / ACCOUNT / TENANT : SOURCE_CONTAINER / SOURCE_OBJECT HTTP/1.1 Destination: TENANT : DEST_CONTAINER / DEST_OBJECT HEAD / API_VERSION / ACCOUNT / TENANT : CONTAINER / OBJECT HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN POST / API_VERSION / ACCOUNT / TENANT : CONTAINER / OBJECT HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN import hmac from hashlib import sha1 from time import time method = 'GET' host = 'https://objectstore.example.com' duration_in_seconds = 300 # Duration for which the url is valid expires = int(time() + duration_in_seconds) path = '/v1/your-bucket/your-object' key = 'secret' hmac_body = '%s\n%s\n%s' % (method, expires, path) hmac_body = hmac.new(key, hmac_body, sha1).hexdigest() sig = hmac.new(key, hmac_body, sha1).hexdigest() rest_uri = "{host}{path}?temp_url_sig={sig}&temp_url_expires={expires}".format( host=host, path=path, sig=sig, expires=expires) print rest_uri https://objectstore.example.com/v1/your-bucket/your-object?temp_url_sig=ff4657876227fc6025f04fcf1e82818266d022c6&temp_url_expires=1423200992 POST / API_VERSION / ACCOUNT HTTP/1.1 Host: FULLY_QUALIFIED_DOMAIN_NAME X-Auth-Token: AUTH_TOKEN https://rgw.domain.com/tenant:container create_container("tenant:container")