How do I make my Apple earphone's play button not launch iTunes and instead pause/play a video on YouTube?
Taken from here: The short answer is that you can't select which media app has priority since they're all equally treated, but if there's nothing playing iTunes takes preference.
The linked answer also pointed to a patch for the Remote Control Daemon at https://github.com/thebitguru/play-button-itunes-patch, which might be what you were looking for. It's a 1 MB download and you will need at least El Capitan (OS 10.11). On that page it says:

This is a patch for removing the default OS X behavior of always starting iTunes when the play button on the keyboard is pressed. This feature can be useful for a lot of users, but it can also be annoying if you are using VLC, Nightingale or other similar programs that support the media keys.
The application will patch the Remote Control Daemon to prevent it from starting iTunes whenever you press the play button on the keyboard or an external remote control. This will only prevent iTunes from starting, all other functions (like play/pause while iTunes is running) will continue to work as before. The original file is backed up in case you would like to restore the original functionality.

It's unfortunately not a straight forward installation. You find the instructions and download on https://thebitguru.com/projects/itunes-patch.