Chapter 7. Management of ceph-immutable-object-cache daemons
Chapter 7. Management of ceph-immutable-object-cache daemons As a storage administrator, use the ceph-immutable-object-cache daemons to cache the parent image content on the local disk. This cache is in the local caching directory. Future reads on that data use the local cache. Figure 7.1. Ceph immutable cache daemon 7.1. Explanation of ceph-immutable-object-cache daemons Cloned Block device images usually modify only a small fraction of the parent image. For example, in a virtual desktop interface (VDI), the virtual machines are cloned from the same base image and initially differ only by the hostname and the IP address. During the bootup, if you use a local cache of the parent image, this speeds up reads on the caching host. This change reduces the client to cluster network traffic. Reasons to use ceph-immutable-object-cache daemons The ceph-immutable-object-cache daemon is a part of Red Hat Ceph Storage. It is a scalable, open-source, and distributed storage system. It connects to local clusters with the RADOS protocol, relying on default search paths to find ceph.conf files, monitor addresses and authentication information for them such as /etc/ceph/ CLUSTER .conf , /etc/ceph/ CLUSTER .keyring , and /etc/ceph/ CLUSTER . NAME .keyring , where CLUSTER is the human-friendly name of the cluster, and NAME is the RADOS user to connect as an example, client.ceph-immutable-object-cache . Key components of the daemon The ceph-immutable-object-cache daemon has the following parts: Domain socket based inter-process communication (IPC): The daemon listens on a local domain socket on start-up and waits for connections from librbd clients. Least recently used (LRU) based promotion or demotion policy: The daemon maintains in-memory statistics of cache-hits on each cache file. It demotes the cold cache if capacity reaches to the configured threshold. File-based caching store: The daemon maintains a simple file based cache store. On promotion the RADOS objects are fetched from RADOS cluster and stored in the local caching directory. When you open each cloned RBD image, librbd tries to connect to the cache daemon through its Unix domain socket. Once successfully connected, librbd coordinates with the daemon on the subsequent reads. If there is a read that is not cached, the daemon promotes the RADOS object to the local caching directory, so the read on that object is serviced from cache. The daemon also maintains simple LRU statistics so that under capacity pressure it evicts cold cache files as needed. Note For better performance, use SSDs as the underlying storage. 7.2. Configuring the ceph-immutable-object-cache daemon The ceph-immutable-object-cache is a daemon for object cache of RADOS objects among Ceph clusters. Important To use the ceph-immutable-object-cache daemon, you must be able to connect RADOS clusters. The daemon promotes the objects to a local directory. These cache objects service the future reads. You can configure the daemon by installing the ceph-immutable-object-cache package. Prerequisites A running Red Hat Ceph Storage cluster. At least one SSD for the cache. Procedure Enable the RBD shared read only parent image cache. Add the following parameters under [client] in the /etc/ceph/ceph.conf file: Example Restart the cluster. Install the ceph-immutable-object-cache package: Example Create a unique Ceph user ID, the keyring: Syntax Example Copy this keyring. In the /etc/ceph directory, create a file and paste the keyring: Example Enable the daemon: Syntax Specify the USER_NAME as the daemon instance. Example Start the ceph-immutable-object-cache daemon: Syntax Example Verification Check the status of the configuration: Syntax Example 7.3. Generic settings of ceph-immutable-object-cache daemons A few important generic settings of ceph-immutable-object-cache daemons are listed. immutable_object_cache_sock Description The path to the domain socket used for communication between librbd clients and the ceph-immutable-object-cache daemon. Type String Default /var/run/ceph/immutable_object_cache_sock immutable_object_cache_path Description The immutable object cache data directory. Type String Default /tmp/ceph_immutable_object_cache immutable_object_cache_max_size Description The maximum size for immutable cache. Type Size Default 1G immutable_object_cache_watermark Description The high-water mark for the cache. The value is between zero and one. If the cache size reaches this threshold the daemon starts to delete cold cache based on LRU statistics. Type Float Default 0.9 7.4. QOS settings of ceph-immutable-object-cache daemons The ceph-immutable-object-cache daemons supports throttling which supports the settings described. immutable_object_cache_qos_schedule_tick_min Description Minimum schedule tick for immutable object cache. Type Milliseconds Default 50 immutable_object_cache_qos_iops_limit Description User-defined immutable object cache IO operations limit per second. Type Integer Default 0 immutable_object_cache_qos_iops_burst Description User-defined burst limit of immutable object cache IO operations. Type Integer Default 0 immutable_object_cache_qos_iops_burst_seconds Description User-defined burst duration in seconds of immutable object cache IO operations. Type Seconds Default 1 immutable_object_cache_qos_bps_limit Description User-defined immutable object cache IO bytes limit per second. Type Integer Default 0 immutable_object_cache_qos_bps_burst Description User-defined burst limit of immutable object cache IO bytes. Type Integer Default 0 immutable_object_cache_qos_bps_burst_seconds Description The desired burst limit of read operations. Type Seconds Default 1
vi /etc/ceph/ceph.conf [client] rbd parent cache enabled = true rbd plugins = parent_cache dnf install ceph-immutable-object-cache ceph auth get-or-create client.ceph-immutable-object-cache. USER_NAME mon 'profile rbd' osd 'profile rbd-read-only' ceph auth get-or-create client.ceph-immutable-object-cache.user mon 'profile rbd' osd 'profile rbd-read-only' [client.ceph-immutable-object-cache.user] key = AQCVPH1gFgHRAhAAp8ExRIsoxQK4QSYSRoVJLw== vi /etc/ceph/ceph.client.ceph-immutable-object-cache.user.keyring [client.ceph-immutable-object-cache.user] key = AQCVPH1gFgHRAhAAp8ExRIsoxQK4QSYSRoVJLw systemctl enable ceph-immutable-object-cache@ceph-immutable-object-cache. USER_NAME systemctl enable ceph-immutable-object-cache@ceph-immutable-object-cache.user Created symlink /etc/systemd/system/ceph-immutable-object-cache.target.wants/ceph-immutable-object-cache@ceph-immutable-object-cache.user.service /usr/lib/systemd/system/ceph-immutable-object-cache@.service. systemctl start ceph-immutable-object-cache@ceph-immutable-object-cache. USER_NAME systemctl start ceph-immutable-object-cache@ceph-immutable-object-cache.user systemctl status ceph-immutable-object-cache@ceph-immutable-object-cache. USER_NAME systemctl status ceph-immutable-object-cache@ceph-immutable-object-cache.user ● ceph-immutable-object-cache@ceph-immutable-object-cache.user> Loaded: loaded (/usr/lib/systemd/system/ceph-immutable-objec> Active: active (running) since Mon 2021-04-19 13:49:06 IST; > Main PID: 85020 (ceph-immutable-) Tasks: 15 (limit: 49451) Memory: 8.3M CGroup: /system.slice/system-ceph\x2dimmutable\x2dobject\x2d> └─85020 /usr/bin/ceph-immutable-object-cache -f --cl>