title: how to solve overlay2 make du report wrong disk size and make shell give no space left on disk error
body: <p>I use the same docker image to launch many containers, now, the /var/lib/docker/overlay2 take up all my disk space, when I login into the shell, it tells me that no space left on the disk, and features like tab completion is disabled.</p>
  
  <p>But, ncdu show that I have a lot of space on disk.</p>
  
  <p>So, what should I do to make the system report the right disk usage?</p>
  
  <pre><code># df -h
  Filesystem                   Size  Used Avail Use% Mounted on
  /dev/mapper/vg_root-lv_root   50G   47G     0 100% /
  devtmpfs                     7.8G     0  7.8G   0% /dev
  tmpfs                        7.8G  8.0K  7.8G   1% /dev/shm
  tmpfs                        7.8G  339M  7.4G   5% /run
  tmpfs                        7.8G     0  7.8G   0% /sys/fs/cgroup
  /dev/sda1                    477M  192M  256M  43% /boot
  /dev/mapper/vg_root-lv_home  860G  637G  180G  79% /home
  tmpfs                        1.6G     0  1.6G   0% /run/user/1000
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/de427a5be70251c986f52d0706d52dee990d70d30f8c2272134be930e4df5e39/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/7e4eaae4e80c19fbb1404dede478a8f71923a6522c7de69289e20183adbad595/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/aaccc3c3715af52b5ee80c85f4bdab6731f6b0432818f2d2d380bff042f854ec/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/c04f9158e166e40ee3fae1c20e11d920f086beccd66bcee15aafcdf43a3c3f14/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/fad78e540d824ae0c773c541b903771544bd4a0e1fa278f6d71cbfeb62b5a9fc/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/e5c2a66f0f0013320592e174341446e03af92bf85aa31b4aa1edfdf7c62947ff/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/9a9d82a83c94fff9a3922609d1a2385b173a06f095b4ae0172711d830b5f6d7c/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/c672e6a835a9203f802a30bb477953bf543fe3d27815c1ca5a9b19a966b60cb6/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/8d2a85881a24fd3dbe2c6b89ca1ca20589e3a9305d2e4ddb509e0606d3ee1624/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/85c9f2a9f740b06fdc3ac2de0e3fd263a0cb99d2e8b82970196c8a779be64aa5/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/faa892c1b6d212ac4fc1278d726f7bef3d55c0314cd266dcabce273138b0c805/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/e04764245c6053839785406dbb3803dc7652f0370816691a26cd0ef52416f9ba/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/7b78bde8e9842f78fa3de8784f1a9d7479c6f0859a8569feb6e397155357d1d9/merged
  shm                           64M  4.0K   64M   1% /var/lib/docker/containers/bcea21f2959df00d92651364b9b1d1fd1592edd76ebc96e74517d27e9781d73c/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/a81222a086a3332812b09215200500801a1bf6e6c6750bb2e4ba8b447984b2b2/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/3c20969a057dd02b683e06e3cec5367b5249ec8179fb360e4cade14ae8428709/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/f585a30576aac3c01e2e013fd575ca0c530b1007708c9a28f18dadd47c2f9fe2/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/c6c35094b2a60960974f8b1551497d12c654ee18125931cb38fd8d04de3547c5/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/376a8bdffeb163b6f6e56b322eaba36ac5ab55db9118ed2a009380d7d2ee210f/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/db781659cd0665b8e23bea94930bf72f0a68e6547a2fa23e5776758d3b277bff/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/9d863f4774b646b797f23d059658ec5fbb263f6298c29342936a719d60d80e26/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/eb518881186826e39b322107257db511c1d57eeefe89872b8d12d0d0c06dbe58/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/fd6ca2f68ee1138d9e89d3f43a58486125560695bad1dda200c023de0562e264/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/0c3c63e10ba60fd6a5339155f48794744b094810bec218de2127ecd2eb777f37/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/61f63c9f8aa9c8bef92c7e59a8af8f8ef23b256e413b813aefc59a5146042639/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/49113505a8b0376f66eaf73c7592e4ebd3bd353b3b787f0b1ad800954c1cdb93/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/506ea7261f670f3375f5aec1242954152b53df076d25442edeebada3d45c26fa/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/8c208146a365fc5e0f37ca62ab43eafc3d39ad83d5255775295aec2e3cd66b59/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/9f79a0563202b2278119735ce49ae72a19d31ce2d61ecc53bdbc77a99fe99e2c/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/b932912cc08e23d740664b90073cf9d1184ef890a757fdc8264229e8e135bf41/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/e976176926d38bfa5465c845c78555f522738be3398fe74dce6db30c710147e4/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/6ff287ddd9f8aea1937954864eaf64db5c6334db0f6edb99655d6d5e479e0868/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/762d362bbf279e1f3d1c94378b6244b6587f8a42f49386913d787ef6cf4d7cc9/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/cfedd98eddf04e6a70ffe4e4677e9706ae767dcc35c046a74ff8684eda0ab721/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/1fffc46bfb192758fe68847831eee74e3b3c7a0f1c83ba9872c40987a33c0011/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/2c975f00d82fefd7a21f9dae494e82579d54d8c1f52ba0d96f1efab2a96682fb/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/70f3735bdba66e10142e421aa5419f55a2b652bafc21d420d2a9cdb6a56d6a9a/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/678032472ea06a749fe2fd67f8490d99228899be7ae38d9dde092f9c89e11893/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/49ba932bb2aac3afb9fc3b03337c8dce1c170f32418716174248e1ae7d67c9c9/merged
  shm                           64M   20K   64M   1% /var/lib/docker/containers/5b9042f0a3dd2e69e29d1aeab2b0e0b2653353382443111e68c89fc7eb2fc85e/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/7dc462026410b4631eb96e0b9e5f0e2b304685226c402370c4747ddce52a08cd/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/a0e8ed4fa8bce5a91620f58326876b3b225b2677139beb78d007e4e745d313aa/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/393e044a1e5beca1231ccc87e519f2c05d4a66f02c29fba0ac0e7215623292b5/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/9d88e58d86ddb8a59e38b6ab23ce3bd0b5830566d8254b350e6ba6cbc8da88fe/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/f762fce5d845b64db036f75fcecbf307cf059a67af34feb4dd9d68046e12f4a3/merged
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/c8a9bbee38a777ac15df037c1987fcba4737ec0b4ab6402406154086ac0615a9/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/e0debd9645d878c23e65e31b395d4828d19ea330452e1a9766d659bee03d334b/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/e44b3d2e5e7c7a341a2f7730db993d6c4367122ea89b9b3be7a76572c05d9a8a/mounts/shm
  shm                           64M   20K   64M   1% /var/lib/docker/containers/c447baa10ed917b49ca27136458ca8f831ef92cd5c0077e336450503eabef86a/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/864c4efbb9deb38a12acc1d8b983be6e3e14ebbfbe4072de99f7cd187a1c773e/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/79d4e8be28c78c082d242614423a28320dc0854da00238811ae7087563c2aeab/mounts/shm
  shm                           64M     0   64M   0% /var/lib/docker/containers/75b69e0f773aa1a8b68e8ac3a0c2f20fee76f6fa8b787085db558f9799463c53/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/99fb3d86e69f54f54b86436be025834a03caf04c0fff81581c4a2f1f3bbed5c4/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/0d15184afd24ef235d95f27bdbb4b7a34523eaf6ae49de8ffdeeb55b2b4a3209/mounts/shm
  overlay                       50G   47G     0 100% /var/lib/docker/overlay2/b57c998741c5d0491cf5805c4a56acf6377aedb1864d519bdd816e51dcaa2de8/merged
  shm                           64M     0   64M   0% /var/lib/docker/containers/912522b1d92b7670627bc7c233be76bfe88c123f7f701acf73fbb42f59d9155a/mounts/shm
  
  
  [root@localhost]# du -hx --max-depth=1 /var/lib/docker/
  16G /var/lib/docker/overlay2
  2.4G    /var/lib/docker/containers
  20K /var/lib/docker/builder
  193M    /var/lib/docker/swarm
  26M /var/lib/docker/image
  4.0K    /var/lib/docker/trust
  4.0K    /var/lib/docker/runtimes
  20K /var/lib/docker/plugins
  22M /var/lib/docker/network
  4.0K    /var/lib/docker/tmp
  72K /var/lib/docker/buildkit
  1.3M    /var/lib/docker/containerd
  216K    /var/lib/docker/volumes
  19G /var/lib/docker/
  
  
  [root@localhost]# du -h --max-depth=1 /var/lib/docker/
  38G /var/lib/docker/overlay2
  2.4G    /var/lib/docker/containers
  20K /var/lib/docker/builder
  193M    /var/lib/docker/swarm
  26M /var/lib/docker/image
  4.0K    /var/lib/docker/trust
  4.0K    /var/lib/docker/runtimes
  20K /var/lib/docker/plugins
  22M /var/lib/docker/network
  4.0K    /var/lib/docker/tmp
  72K /var/lib/docker/buildkit
  1.3M    /var/lib/docker/containerd
  216K    /var/lib/docker/volumes
  40G /var/lib/docker/
  </code></pre>
  
  <p>when press tab in shell:</p>
  
  <pre><code>-bash: cannot create temp file for here-document: No space left on device
  </code></pre>