During weekend my Debian based proxy crashed due to insufficient free space. After reboot it was fine again, so on monday I went hunting for logs and/or explanations for saturday hang but I couldn't find anything.Grepped any kind of stuff in /var/log/*, checked crontab, mail queue... The only thing I found was monitoring system echos in syslog, with free space getting critical minute after minute (filled up about 80GB in 30mins). No daemons error or so.I would easily understand what is going on in realtime, but have no idea on how to further debug this kind of problem in the past. Any advice?This is the first time this has happened in a year or so. Uptime was low, problem did not reoccur in next days.Thank you
How to detect which process filled up disk space
process;monitoring;disk usage
You can use atop to debug stuff like this. atop can run in realtime, but more critically for your case, it can show and analyse snapshots from the past captured in logs. It logs a lot of different metrics, so you likely won't be left thinking damn, I wish I logged that after the fact. :-)On Debian, you can install it with apt-get install atop. You can then start and enable it on boot using your init manager. On systemd for example, it would be systemctl enable atop && systemctl start atop. Now atop will start logging -- typically this is to /var/log/atop/<date>.You can view historic logs by using atop -r <log file>, going forward in time with t, and backwards with T. You can find more commands by pressing the ? key.You should look for an app writing a lot to the disk. You can see this in the WRDISK column. You can also sort by disk usage by pressing D.Obviously this can't go back to before atop even started logging, but you can have it running and logging in the background for next time, when you can investigate properly.