How to load library only on one twig template?
Just remove the reference from mytheme.info.yml. 

That's adding it any time the theme is loaded (every page), whereas attach_library will add it only when that specific template is used.

You'll need to rebuild cache for the change to take effect.