title: Why Ceph turns status to Err when there is still available storage space
body: <p>I built a 3 node Ceph cluster recently. Each node had seven 1TB HDD for OSDs. In total, I have 21 TB of storage space for Ceph. </p>
  
  <p>However, when I ran a workload to keep writing data to <code>Ceph</code>, it turns to <code>Err</code> status and no data can be written to it any more. </p>
  
  <p>The output of <code>ceph -s</code> is:  </p>
  
  <pre><code> cluster:
      id:     06ed9d57-c68e-4899-91a6-d72125614a94
      health: HEALTH_ERR
              1 full osd(s)
              4 nearfull osd(s)
              7 pool(s) full
  
    services:
      mon: 1 daemons, quorum host3
      mgr: admin(active), standbys: 06ed9d57-c68e-4899-91a6-d72125614a94
      osd: 21 osds: 21 up, 21 in
      rgw: 4 daemons active
  
    data:
      pools:   7 pools, 1748 pgs
      objects: 2.03M objects, 7.34TiB
      usage:   14.7TiB used, 4.37TiB / 19.1TiB avail
      pgs:     1748 active+clean
  </code></pre>
  
  <p>Based on my comprehension, since there is still 4.37 TB space left, <code>Ceph</code> itself should take care about how to balance the workload and make each OSD to not be at <code>full</code> or <code>nearfull</code> status. But the result doesn't work as my expectation, <code>1 full osd</code> and <code>4 nearfull osd</code> shows up, the health is <code>HEALTH_ERR</code>.</p>
  
  <p>I can't visit Ceph with <code>hdfs</code> or <code>s3cmd</code> anymore, so here comes the question:<br>
  1, Any explanation about current issue?<br>
  2, How can I recover from it? Delete data on Ceph node directly with ceph-admin, and relaunch the Ceph?</p>