anchor.diff:  ----
   apiVersion: v1
   kind: PersistentVolume
  -metadata:
  -  name: "ceph-pv" <1>
  -spec:
  +metadata: 
  +  name: ceph-pv <1>
  +spec: 
     capacity:
  -    storage: "2Gi" <2>
  +    storage: 2Gi <2>
     accessModes:
  -    - "ReadWriteOnce" <3>
  +    - ReadWriteOnce <3>
     rbd: <4>
  -    monitors:
  -      - "192.168.122.111:6789" <5>
  +    monitors: <5>
  +      - 192.168.122.133:6789
       pool: rbd
  -    image: foo
  +    image: ceph-image
       user: admin
  -    secretRef:
  -      name: "ceph-secret" <6>
  -    fsType: ext4
  +    secretRef: 
  +      name: ceph-secret <6>
  +    fsType: ext4 <7>
       readOnly: false
  -  persistentVolumeReclaimPolicy: "Recycle"
  -----
  -<1> The name of the volume. This will be how it is identified via
  -link:../../architecture/additional_concepts/storage.html[persistent volume
  -claims] or from pods.
  +  persistentVolumeReclaimPolicy: Recycle
  +----
  +<1> The name of the PV which is referenced in pod definitions or displayed in
  +various `oc` volume commands.
   <2> The amount of storage allocated to this volume.
  -<3> This is the access type to the block storage. All block storage is defined to be single user (non-shared storage).
  -<4> This defines the volume type being used, in this case the *rbd*
  -plug-in.
  -<5> This is the Ceph monitor address and port.
  -<6> This is the Ceph secret used to create a secure connection from OSE to the Ceph server. This must be configured ahead of time
  -as part of the Ceph configuration.
  +<3> `accessModes` are used as labels to match a PV and a PVC. They currently
  +do not define any form of access control. All block storage is defined to be
  +single user (non-shared storage).
  +<4> This defines the volume type being used, in this case the *rbd* plug-in.
  +<5> This is an array of Ceph monitor ip addresses and ports.
  +<6> This is the Ceph secret, defined above, used to create a secure connection
  +from OpenShift to the Ceph server.
  +<7> This is the file system type mounted on the Ceph RBD block device.
   ====
   
   Save your definition to a file, for example *_ceph-pv.yaml_*, and create the
  
positive.code: toc::[]
  
  == Overview
  OpenShift can utilize persistent storage using Block Storage
  like Ceph RBD. Some familiarity with Kubernetes and Docker is assumed. It is
  also assumed that there is access to an existing Ceph cluster and volume,
  and that *ceph-common* package has been installed on all OpenShift nodes in the
  cluster.
  
  The Kubernetes
  link:../../architecture/additional_concepts/storage.html[persistent volume]
  framework allows administrators to provision a cluster with persistent storage
  and gives users a way to request those resources without having any knowledge of
  the underlying infrastructure. Persistent volumes are not bound to a single
  project or namespace; they can be shared across the OpenShift cluster.
  link:../../architecture/additional_concepts/storage.html#persistent-volume-claims[Persistent
  volume claims], however, are specific to a project or namespace and can be
  requested by users.
  
  [IMPORTANT]
  ====
  
negative.code: ----
  ====
  
  Developers request Ceph-RBD storage by referencing, in the `*volumes*` section
  of their pod spec, either a
  link:../../dev_guide/persistent_volumes.html[`*PersistentVolumeClaim*`] by name,
  or the gluster volume plug-in directly. A PVC exists only in the user's namespace
  and can only be referenced only by pods within that same namespace. Any attempt to
  access a PV across a namespace causes the pod to fail.
  
  The next step can be to create a persistent volume claim (PVC) which will bind
  to the new PV:
  
  .PVC Object Definition
  ====
  [source,yaml]
  ----
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: ceph-claim
  spec:
    accessModes: <1>
      - ReadWriteOnce
    resources:
      requests:
        storage: 2Gi <2>
  
  ----
  <1> As mentioned above for PVs, the `*accessModes*` do not enforce access right,
  but rather act as labels to match a PV to a PVC.
  <2> This claim will look for PVs offering *2Gi* or greater capacity.
  ====
  
  Save the definition to a file, for example *_ceph-claim.yaml_*, and create the
  PVC:
  
  ====
  ----
  # oc create -f ceph-claim.yaml
  ----
  ====
  
  [[ceph-volume-security]]
  == Volume Security
  
  This section covers Ceph-RBD block volume security.
  
  [NOTE]
  ====
  See
  link:../../install_config/persistent_storage/pod_security_context#fsgroup.html[Volume
  Security] before implementing Ceph RBD volumes.
  ====
  
  A significant difference between *shared* volumes (such as NFS and GlusterFS) versus
  *block* volumes (such as Ceph-RBD, iSCSI, and most cloud storage) is that the user and
  group IDs defined in the pod definition (or in the docker image) are applied to the
  *_target_* (physical) storage. This is referred to as *_managing ownership_* of the
  block device.  In other words, if the real Ceph RBD mount has its owner set to *123*
  and its group id set to *567*, and if the pod defines its `runAsUser` equal to *222*
  and its `fsGroup` to be *7777*, then the Ceph RBD physical mount's ownership will be
  *_changed_* to *222:7777*.
  
  [NOTE]
  ====
  Even if the user and group ids are not defined in the pod spec, the resulting pod may
  have defaults defined for these ids based on its matching SCC or its project. The
  link:../../install_config/persistent_storage/pod_security_context[pod security section]
  covers storage aspects of SCCs and defaults in greater detail.
  ====
  
  A pod defines the group ownership of a Ceph RBD volume using the `*fsGroup*` stanza under
  the pod's `*securityContext*` definition.  For example:
  
  ====
  ----
  spec:
    containers:
      - name:
      ...
    securityContext: <1>
      fsGroup: 7777 <2>
  ----
  <1> `*securityContext*` must be defined at the pod level, not under a specific container.
  <2> all containers in the pod will have the same fsGroup id.
  ====
  