Stream different videos and sounds to Bluetooth headphones & loudspeakers
OS X itself supports each app using a different device, as long as that app support select a device instead of using the default one.
Unfortunately neither iTunes or Safari supports alternative device unless that's an AirPlay device.

You have still two options here:


get an AirPlay enabled device, so you may choose device in iTunes or Youtube Player (which is powered by HTML 5 not Flash)
get another music app other than iTunes, I'd recommend Vox for Mac[1] which supports play iTunes Library and a lot more, you may choose device in the app. 


UPDATE 2016/02/20
AirFoil 5 for Mac[2] now supports set different audio source for different apps.

[1] http://coppertino.com/vox/mac
[2] https://rogueamoeba.com/airfoil/mac/