title: AWS EC2 disk full, cannot release space
body: <p>I am using an EC2 instance to run a node app. I logged into the server after a while only to realise that the server has run out of disk space. After debugging, I realised that logs are taking up space. I deleted the 3.3Gb log file. However, even after the cleanup there is no space. What should I do?</p>
  
  <p>Here are the commands I ran:</p>
  
  <pre><code>ubuntu@app1:~$ df -h
  Filesystem      Size  Used Avail Use% Mounted on
  udev            488M     0  488M   0% /dev
  tmpfs           100M   11M   89M  11% /run
  /dev/xvda1      7.7G  7.7G     0 100% /
  tmpfs           496M  8.0K  496M   1% /dev/shm
  tmpfs           5.0M     0  5.0M   0% /run/lock
  tmpfs           496M     0  496M   0% /sys/fs/cgroup
  tmpfs           100M     0  100M   0% /run/user/1001
  tmpfs           100M     0  100M   0% /run/user/1000
  </code></pre>
  
  <hr>
  
  <pre><code>ubuntu@app1:~$ sudo du -h --max-depth=1 / | sort -n
  0   /proc
  0   /sys
  4.0K    /lib64
  4.0K    /media
  4.0K    /mnt
  4.0K    /srv
  8.0K    /dev
  8.0K    /snap
  16K /lost+found
  24K /root
  800K    /tmp
  6.4M    /etc
  11M /run
  14M /sbin
  16M /bin
  246M    /boot
  331M    /home
  397M    /opt
  429M    /var
  538M    /lib
  2.1G    /usr
  3.7G    /data
  7.7G    /
  </code></pre>
  
  <p>I deleted a 3.3G log file in <code>/data</code> and ran <code>du</code> again</p>
  
  <pre><code>ubuntu@app1:~$ sudo du -h --max-depth=1 / | sort -h
  0   /proc
  0   /sys
  4.0K    /lib64
  4.0K    /media
  4.0K    /mnt
  4.0K    /srv
  8.0K    /dev
  8.0K    /snap
  16K /lost+found
  24K /root
  800K    /tmp
  6.4M    /etc
  11M /run
  14M /sbin
  16M /bin
  246M    /boot
  331M    /home
  352M    /data
  397M    /opt
  429M    /var
  538M    /lib
  2.1G    /usr
  4.4G    /
  </code></pre>
  
  <hr>
  
  <pre><code>ubuntu@app1:~$ df -h
  Filesystem      Size  Used Avail Use% Mounted on
  udev            488M     0  488M   0% /dev
  tmpfs           100M   11M   89M  11% /run
  /dev/xvda1      7.7G  7.7G     0 100% /
  tmpfs           496M  8.0K  496M   1% /dev/shm
  tmpfs           5.0M     0  5.0M   0% /run/lock
  tmpfs           496M     0  496M   0% /sys/fs/cgroup
  tmpfs           100M     0  100M   0% /run/user/1001
  tmpfs           100M     0  100M   0% /run/user/1000
  </code></pre>
  
  <p><strong>Although the <code>/data</code> directory is now reduced to 352M, still <code>df</code> still shows 100% disk utilization. What am I missing?</strong></p>
  
  <p>Referring to this answer <a href="https://unix.stackexchange.com/a/253655/47050">https://unix.stackexchange.com/a/253655/47050</a>, here is the output of <code>strace</code></p>
  
  <pre><code>ubuntu@app1:~$ strace -e statfs df /
  statfs("/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2016361, f_bfree=4096, f_bavail=0, f_files=1024000, f_ffree=617995, f_fsid={2136106470, -680157247}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
  Filesystem     1K-blocks    Used Available Use% Mounted on
  /dev/xvda1       8065444 8049060         0 100% /
  +++ exited with 0 +++
  </code></pre>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I ran </p>
  
  <p><code>sudo lsof | grep deleted</code></p>
  
  <p>and found</p>
  
  <pre><code>node\x20/ 22318           deploy   12w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
  node\x20/ 22318           deploy   13w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
  node\x20/ 22318           deploy   14w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
  node\x20/ 22318           deploy   15w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
  node\x20/ 22318           deploy   16w      REG              202,1 3541729280     791684 /data/app/shared/logs/production.log (deleted)
  </code></pre>
  
  <p>How do I release these files?</p>
  
  <p><strong>Update 2</strong></p>
  
  <pre><code>ubuntu@app1:~$ sudo ls -l /proc/22318/fd
  total 0
  lrwx------ 1 deploy deploy 64 Apr  6 10:05 0 -&gt; socket:[74749956]
  lrwx------ 1 deploy deploy 64 Apr  6 10:05 1 -&gt; socket:[74749958]
  lr-x------ 1 deploy deploy 64 Apr  6 10:05 10 -&gt; /dev/null
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 12 -&gt; /data/app/shared/logs/production.log (deleted)
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 13 -&gt; /data/app/shared/logs/production.log (deleted)
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 14 -&gt; /data/app/shared/logs/production.log (deleted)
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 15 -&gt; /data/app/shared/logs/production.log (deleted)
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 16 -&gt; /data/app/shared/logs/production.log (deleted)
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 17 -&gt; /data/app/shared/logs/production.log (deleted)
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 18 -&gt; /data/app/shared/logs/production.log (deleted)
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 19 -&gt; /data/app/shared/logs/production.log (deleted)
  lrwx------ 1 deploy deploy 64 Apr  6 10:05 2 -&gt; socket:[74749960]
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 20 -&gt; /data/app/shared/logs/production.log (deleted)
  lrwx------ 1 deploy deploy 64 Apr  6 10:05 21 -&gt; socket:[74750302]
  lrwx------ 1 deploy deploy 64 Apr  6 10:05 22 -&gt; socket:[74750303]
  lrwx------ 1 deploy deploy 64 Apr  6 10:05 3 -&gt; socket:[74749962]
  lrwx------ 1 deploy deploy 64 Apr  6 10:05 4 -&gt; anon_inode:[eventpoll]
  lr-x------ 1 deploy deploy 64 Apr  6 10:05 5 -&gt; pipe:[74749978]
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 6 -&gt; pipe:[74749978]
  lr-x------ 1 deploy deploy 64 Apr  6 10:05 7 -&gt; pipe:[74749979]
  l-wx------ 1 deploy deploy 64 Apr  6 10:05 8 -&gt; pipe:[74749979]
  lrwx------ 1 deploy deploy 64 Apr  6 10:05 9 -&gt; anon_inode:[eventfd]
  </code></pre>
  
  <hr>
  
  <pre><code>ubuntu@app1:~$ ps aux | grep node
  deploy   22318  0.0 12.7 1277192 129844 ?      Ssl   2019 173:38 node /data/app/releases/20180904094535/app.js
  ubuntu   30665  0.0  0.0  12944   972 pts/0    S+   10:09   0:00 grep --color=auto node
  </code></pre>