anchor.diff:  ----
   ====
   
  -Users can then link:../../dev_guide/persistent_volumes.html[request storage
  -using persistent volume claims], which can now utilize your new persistent
  -volume.
  -
  -[IMPORTANT]
  -====
  -Persistent volume claims only exist in the user's namespace and can only be
  -referenced by a pod within that same namespace. Any attempt to access a
  -persistent volume from a different namespace causes the pod to fail.
  -====
  +Developers request Ceph-RBD storage by referencing, in the `*volumes*` section
  +of their pod spec, either a
  +link:../../dev_guide/persistent_volumes.html[`*PersistentVolumeClaim*`] by name,
  +or the gluster volume plug-in directly. A PVC exists only in the user's namespace
  +and can only be referenced only by pods within that same namespace. Any attempt to
  +access a PV across a namespace causes the pod to fail.
  +
  +The next step can be to create a persistent volume claim (PVC) which will bind
  +to the new PV:
  +
  +.PVC Object Definition
  +====
  +[source,yaml]
  +----
  +kind: PersistentVolumeClaim
  +apiVersion: v1
  +metadata:
  +  name: ceph-claim
  +spec:
  +  accessModes: <1>
  +    - ReadWriteOnce
  +  resources:
  +    requests:
  +      storage: 2Gi <2>
  +
  +----
  +<1> As mentioned above for PVs, the `*accessModes*` do not enforce access right,
  +but rather act as labels to match a PV to a PVC.
  +<2> This claim will look for PVs offering *2Gi* or greater capacity.
  +====
  +
  +Save the definition to a file, for example *_ceph-claim.yaml_*, and create the
  +PVC:
  +
  +====
  +----
  +# oc create -f ceph-claim.yaml
  +----
  +====
  +
  +[[ceph-volume-security]]
  +== Volume Security
  +
  +This section covers Ceph-RBD block volume security.
  +
  +[NOTE]
  +====
  +See
  +link:../../install_config/persistent_storage/pod_security_context#fsgroup.html[Volume
  +Security] before implementing Ceph RBD volumes.
  +====
  +
  +A significant difference between *shared* volumes (such as NFS and GlusterFS) versus
  +*block* volumes (such as Ceph-RBD, iSCSI, and most cloud storage) is that the user and
  +group IDs defined in the pod definition (or in the docker image) are applied to the
  +*_target_* (physical) storage. This is referred to as *_managing ownership_* of the
  +block device.  In other words, if the real Ceph RBD mount has its owner set to *123*
  +and its group id set to *567*, and if the pod defines its `runAsUser` equal to *222*
  +and its `fsGroup` to be *7777*, then the Ceph RBD physical mount's ownership will be
  +*_changed_* to *222:7777*.
  +
  +[NOTE]
  +====
  +Even if the user and group ids are not defined in the pod spec, the resulting pod may
  +have defaults defined for these ids based on its matching SCC or its project. The
  +link:../../install_config/persistent_storage/pod_security_context[pod security section]
  +covers storage aspects of SCCs and defaults in greater detail.
  +====
  +
  +A pod defines the group ownership of a Ceph RBD volume using the `*fsGroup*` stanza under
  +the pod's `*securityContext*` definition.  For example:
  +
  +====
  +----
  +spec:
  +  containers:
  +    - name:
  +    ...
  +  securityContext: <1>
  +    fsGroup: 7777 <2>
  +----
  +<1> `*securityContext*` must be defined at the pod level, not under a specific container.
  +<2> all containers in the pod will have the same fsGroup id.
  +====
  
positive.code: ====
  
  [[ceph-provisioning]]
  
  == Provisioning
  Storage must exist in the underlying infrastructure before it can be mounted as
  a volume in OpenShift. To provision Ceph volumes in OpenShift, the
  following are required:
  
  - Existing Ceph volumes, to be defined in the persistent volume object
  - The `*PersistentVolume*` API
  
  You must also ensure *ceph-common* package is installed on all OpenShift nodes in the cluster:
  
  ----
  # yum install ceph-common
  ----
  
  
  [[ceph-creating-persistent-volume]]
  
  === Creating the Persistent Volume
  
  You must define your persistent volume in an object definition before creating
  it in OpenShift:
  
  .Persistent Volume Object Definition Using Ceph RBD
  
negative.code: ----
  apiVersion: v1
  kind: PersistentVolume
  metadata: 
    name: ceph-pv <1>
  spec: 
    capacity:
      storage: 2Gi <2>
    accessModes:
      - ReadWriteOnce <3>
    rbd: <4>
      monitors: <5>
        - 192.168.122.133:6789
      pool: rbd
      image: ceph-image
      user: admin
      secretRef: 
        name: ceph-secret <6>
      fsType: ext4 <7>
      readOnly: false
    persistentVolumeReclaimPolicy: Recycle
  ----
  <1> The name of the PV which is referenced in pod definitions or displayed in
  various `oc` volume commands.
  <2> The amount of storage allocated to this volume.
  <3> `accessModes` are used as labels to match a PV and a PVC. They currently
  do not define any form of access control. All block storage is defined to be
  single user (non-shared storage).
  <4> This defines the volume type being used, in this case the *rbd* plug-in.
  <5> This is an array of Ceph monitor ip addresses and ports.
  <6> This is the Ceph secret, defined above, used to create a secure connection
  from OpenShift to the Ceph server.
  <7> This is the file system type mounted on the Ceph RBD block device.
  ====
  
  Save your definition to a file, for example *_ceph-pv.yaml_*, and create the
  