title: health: HEALTH_ERR - how to fix it without losing data?
body: <p>got ceph status:</p>
  
  <pre><code># ceph status
    cluster:
      id:     b683c5f1-fd15-4805-83c0-add6fbb7faae
      health: HEALTH_ERR
              1 backfillfull osd(s)
              8 pool(s) backfillfull
              50873/1090116 objects misplaced (4.667%)
              Degraded data redundancy: 34149/1090116 objects degraded (3.133%), 3 pgs degraded, 3 pgs undersized
              Degraded data redundancy (low space): 6 pgs backfill_toofull
  
    services:
      mon: 3 daemons, quorum tb-ceph-2-prod,tb-ceph-4-prod,tb-ceph-3-prod
      mgr: tb-ceph-1-prod(active)
      osd: 6 osds: 6 up, 6 in; 6 remapped pgs
      rgw: 4 daemons active
  
    data:
      pools:   8 pools, 232 pgs
      objects: 545.1 k objects, 153 GiB
      usage:   728 GiB used, 507 GiB / 1.2 TiB avail
      pgs:     34149/1090116 objects degraded (3.133%)
               50873/1090116 objects misplaced (4.667%)
               226 active+clean
               3   active+undersized+degraded+remapped+backfill_toofull
               3   active+remapped+backfill_toofull
  
    io:
      client:   286 KiB/s rd, 2 op/s rd, 0 op/s wr
  </code></pre>
  
  <p>Here is OSD statuses:</p>
  
  <pre><code># ceph osd df
  ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS
   2   hdd 0.09769  1.00000 100 GiB  32 GiB  68 GiB 32.38 0.55  30
   5   hdd 0.32230  1.00000 330 GiB 220 GiB 110 GiB 66.71 1.13 122
   0   hdd 0.32230  1.00000 330 GiB 194 GiB 136 GiB 58.90 1.00 125
   1   hdd 0.04390  0.95001  45 GiB  43 GiB 2.5 GiB 94.53 1.60  11
   3   hdd 0.09769  1.00000 100 GiB  42 GiB  58 GiB 42.37 0.72  44
   4   hdd 0.32230  0.95001 330 GiB 196 GiB 134 GiB 59.43 1.01 129
                      TOTAL 1.2 TiB 728 GiB 507 GiB 58.94
  MIN/MAX VAR: 0.55/1.60  STDDEV: 19.50
  </code></pre>
  
  <p>I have tried these commands:</p>
  
  <pre><code> ceph osd pool set default.rgw.buckets.data pg_num 32
   ceph osd pool set default.rgw.buckets.data pgp_num 32
  </code></pre>
  
  <p>But it didn't help either. I think pg_num 32 is too small for my OSD count, but not sure if it's safe to set it bigger while health status is err</p>