How to set cache meta data on drupalSettings?
When you add attachments you can add cache data as well: 

$attachments['#cache']['tags'][] = 'config:my_module.settings';


BTW #attached and #cache are the only data you are allowed to add in hook_page_attachments() or hook_page_attachments_alter().