Adding a cache tag to an entity view display doesn't do anything?
For that to work, core would have to use getCacheTags() from that config entity, which it doesn't do.

I'm not sure if that should be changed, core currently does not add the cache tags of that entity on purpose, instead it it invalidates the generic {$entity_type}_ cache tag on saving such an entity in \Drupal\Core\Entity\Entity\EntityViewDisplay::postSave(). Consdering that this changes only very infrequently once a site is live, this is more performant as there are fewer cache tags to check on each request.