Cache context drupal don't work
Anonymous page cache does not respect cache contexts, I'm guessing that is your problem.

To get that working, you need to switch out the PageCache service in a custom module with a ServiceProvider and override the getCacheId() method.