anchor.diff:  ====
   
   [[ceph-provisioning]]
  -
   == Provisioning
  +
   Storage must exist in the underlying infrastructure before it can be mounted as
  -a volume in OpenShift. To provision Ceph volumes in OpenShift, the
  -following are required:
  -
  -- Existing Ceph volumes, to be defined in the persistent volume object
  -- The `*PersistentVolume*` API
  -
  -You must also ensure *ceph-common* package is installed on all OpenShift nodes in the cluster:
  +a volume in OpenShift. To provision Ceph volumes in OpenShift, the following
  +information is required:
  +
  +- The ceph key to be used in an OpenShift secret object.
  +- The ceph image name.
  +- The file system type on top of the block storage (e.g., ext4).
  +
  +You must also ensure that the *ceph-common* package is installed on *_all_*
  +schedulable OpenShift nodes:
   
   ----
   # yum install ceph-common
   ----
   
  -
  -[[ceph-creating-persistent-volume]]
  -
  +[[creating-ceph-secret]]
  +=== Creating the Ceph Secret
  +
  +In a secret definition, the storage administrator must define the admin authorization key,
  +which is then converted to base-64 for use by OpenShift. The `ceph auth get-key` command
  +is run on a Ceph MON node to display the key value for the _client.admin_ user:
  +
  +.Ceph Secret Definition
  +====
  +[source,yaml]
  +----
  +apiVersion: v1
  +kind: Secret
  +metadata:
  +  name: ceph-secret
  +data:
  +  key: QVFBOFF2SlZheUJQRVJBQWgvS2cwT1laQUhPQno3akZwekxxdGc9PQ== <1>
  +
  +----
  +<1> this base64 key is generated on one of the Ceph MON nodes using the following command:
  +  `ceph auth get-key client.admin | base64`
  +and then copying the output and pasting it as the secret key's value.
  +====
  +
  +Save the secret definition to a file, for example *_ceph-secret.yaml_*,
  +then create the secret:
  +
  +====
  +----
  +$ oc create -f ceph-secret.yaml
  +secret "ceph-secret" created
  +----
  +====
  +
  +Verify that the secret was created:
  +
  +====
  +----
  +# oc get secret ceph-secret
  +NAME          TYPE      DATA      AGE
  +ceph-secret   Opaque    1         23d
  +----
  +====
  +
  +[[ceph-creating-pv]]
   === Creating the Persistent Volume
   
  -You must define your persistent volume in an object definition before creating
  +The storage admin next defines the persistent volume (PV) in an object definition before creating
   it in OpenShift:
   
   .Persistent Volume Object Definition Using Ceph RBD
  
positive.code: ----
  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: "ceph-pv" <1>
  spec:
    capacity:
      storage: "2Gi" <2>
    accessModes:
      - "ReadWriteOnce" <3>
    rbd: <4>
      monitors:
        - "192.168.122.111:6789" <5>
      pool: rbd
      image: foo
      user: admin
      secretRef:
        name: "ceph-secret" <6>
      fsType: ext4
      readOnly: false
    persistentVolumeReclaimPolicy: "Recycle"
  ----
  <1> The name of the volume. This will be how it is identified via
  link:../../architecture/additional_concepts/storage.html[persistent volume
  claims] or from pods.
  <2> The amount of storage allocated to this volume.
  <3> This is the access type to the block storage. All block storage is defined to be single user (non-shared storage).
  <4> This defines the volume type being used, in this case the *rbd*
  plug-in.
  <5> This is the Ceph monitor address and port.
  <6> This is the Ceph secret used to create a secure connection from OSE to the Ceph server. This must be configured ahead of time
  as part of the Ceph configuration.
  ====
  
  Save your definition to a file, for example *_ceph-pv.yaml_*, and create the
  
negative.code: ----
  ====
  
  Developers request Ceph-RBD storage by referencing, in the `*volumes*` section
  of their pod spec, either a
  link:../../dev_guide/persistent_volumes.html[`*PersistentVolumeClaim*`] by name,
  or the gluster volume plug-in directly. A PVC exists only in the user's namespace
  and can only be referenced only by pods within that same namespace. Any attempt to
  access a PV across a namespace causes the pod to fail.
  
  The next step can be to create a persistent volume claim (PVC) which will bind
  to the new PV:
  
  .PVC Object Definition
  ====
  [source,yaml]
  ----
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: ceph-claim
  spec:
    accessModes: <1>
      - ReadWriteOnce
    resources:
      requests:
        storage: 2Gi <2>
  
  ----
  <1> As mentioned above for PVs, the `*accessModes*` do not enforce access right,
  but rather act as labels to match a PV to a PVC.
  <2> This claim will look for PVs offering *2Gi* or greater capacity.
  ====
  
  Save the definition to a file, for example *_ceph-claim.yaml_*, and create the
  PVC:
  
  ====
  ----
  # oc create -f ceph-claim.yaml
  ----
  ====
  
  [[ceph-volume-security]]
  == Volume Security
  
  This section covers Ceph-RBD block volume security.
  
  [NOTE]
  ====
  See
  link:../../install_config/persistent_storage/pod_security_context#fsgroup.html[Volume
  Security] before implementing Ceph RBD volumes.
  ====
  
  A significant difference between *shared* volumes (such as NFS and GlusterFS) versus
  *block* volumes (such as Ceph-RBD, iSCSI, and most cloud storage) is that the user and
  group IDs defined in the pod definition (or in the docker image) are applied to the
  *_target_* (physical) storage. This is referred to as *_managing ownership_* of the
  block device.  In other words, if the real Ceph RBD mount has its owner set to *123*
  and its group id set to *567*, and if the pod defines its `runAsUser` equal to *222*
  and its `fsGroup` to be *7777*, then the Ceph RBD physical mount's ownership will be
  *_changed_* to *222:7777*.
  
  [NOTE]
  ====
  Even if the user and group ids are not defined in the pod spec, the resulting pod may
  have defaults defined for these ids based on its matching SCC or its project. The
  link:../../install_config/persistent_storage/pod_security_context[pod security section]
  covers storage aspects of SCCs and defaults in greater detail.
  ====
  
  A pod defines the group ownership of a Ceph RBD volume using the `*fsGroup*` stanza under
  the pod's `*securityContext*` definition.  For example:
  
  ====
  ----
  spec:
    containers:
      - name:
      ...
    securityContext: <1>
      fsGroup: 7777 <2>
  ----
  <1> `*securityContext*` must be defined at the pod level, not under a specific container.
  <2> all containers in the pod will have the same fsGroup id.
  ====
  