Dynamically remove library based on user role, cache context not working
hook_library_info_alter() is not the right approach. You get the role of the user who installs a module or clears the cache, not of the user who requests the page. Info alter hooks in general are about changing information provided by other extensions and usually are only recalculated if an extension is added or removed and for a better dev experience on a cache clear. 

And indeed, hook_page_attachments_alter() lists only libraries added by other hook_page_attachments(_alter) hooks. See invokePageAttachmentHooks(), which starts with an empty attachments array:

  public function invokePageAttachmentHooks(array &amp;$page) {
    // Modules can add attachments.
    $attachments = [];
    ...


So we can rule out both hooks you've mentioned. Attached libraries are collected from many different places (YAML, render arrays, hooks) and they are merged (bubbled up) through the entire page building and rendering process. So it might be a good idea to attach the override in form of a custom library the same  way and let Drupal calculate the assets when rendering the page. See https://www.drupal.org/node/2216195#override-extend