How can I "outsmart" caching?
The issue you have is with the module Internal Page Cache for anonymous users. You have several options:

If you want to serve for anonymous users a newly built page on each request the same way as for logged in users, then uninstall the Internal Page Cache module. The dynamic page cache will still cache pages, but does respect the cache contexts and max-age you set on the page elements. 

Performance wise it is not a good idea to do this, so it might be better to configure a max-age for the Internal Page Cache, see this answer https://drupal.stackexchange.com/a/196557/47547, where you use an event subscriber to set an Expires time in the response.

If this is only for Views, another option is to modify the route definition of the view by setting a no_cache option, see this answer https://drupal.stackexchange.com/a/230242/47547

In all cases you need to configure /admin/config/development/performance for external caches like the browser cache.