Security test generated large compiled_templates directory, site outage
I don't have specific answers to your questions, but that's mainly because I feel there are some clarifications and observations that I think might be more helpful.

The thread you linked to is referring to template caching using the Craft {% cache %} tag in a template.  All of its cached data is stored in the database in the craft_templatecache* tables, not on the file system.  See here for an explanation of the different caching methods Craft uses.

The compiled files in craft/storage/runtime/compiled_templates are what happens when Twig needs to render a template in the craft/templates folder. The first time a template is requested, Twig will take that uncompiled template and compile it down to PHP and save it in the compiled_templates folder.  If another request comes in for that same template, it will check the  file's last modified date and if it hasn't changed, it will execute the compiled template directly.

Given that an actual 404 request wouldn't create new files in the compiled_templates folder (other than the 404 template that it might need to render), I'd guess that the assumption that the 404 requests caused the compiled_templates folder to fill up enough to run out of disk space is incorrect.  It could be that enough log files or some other by-product of the testing caused more disk space to be used, but it's hard to tell without looking at the server.

You also mentioned that the database went down.  Given that the {% cache %} tag stores its data in the database, my first hunch would be to look at the strategy for using the template cache tag on your templates.  Here is a good article that weighs the in-and-outs of using it: https://nystudio107.com/blog/the-craft-cache-tag-in-depth

Two other things worth thinking about. A responsible security pen-tester is going to know better than to test on a live site by hammering it to the ground. But it's possible that maybe they were being responsible and the server is under-powered to handle any spike in load.