Large Library/Containers directory
Go to ~/Library/Containers/ and type du -h command. Now you will be more able to see which folder/file is taking most of the space in the system. My main culprit was Docker.