Store cache's datas in files instead of database
Drupal comes with a php-file based cache backend by default: \Drupal\Core\Cache\PhpBackend, the factory is available as the cache.backend.php service.

You would configure that to use that by default with $settings['cache']['default'] = 'cache.backend.php';.

That said, I wouldn't really recommend using that. PHP files are going to fill your disk as well (without a cleanup-mechanism that the database now has) and if you have a very large site then you probably also have multiple servers and then cache invalidation becomes a problem again, clearing the cache on one server won't clear it on others unless you use a shared file system and that is again slow.

The recommended approach IMHO for a big site is to use an in-memory cache backend like Redis or Memcache (varnish is not a cache backend that is used by Drupal, it's something that sits in front of it). The big advantage of those two is that they store their data only stored in memory (configurable in case of redis) and very fast to access due to that but still easy to for multiple servers. They also have built-in purging of data and a fixed size, so they will automatically delete less-frequently used cache entries. And while not perfect, they are far better at that than the default database backend now which just deletes old cache entries and keeps a fixed amount of caches.