Youtube videos open in browser by default
Try resetting app preferences. Go to Settings->Apps, choose from menu Reset app preferences and confirm Reset apps. This will reset all app preferences.

To reset preferences for specific app (e.g. Browser), choose it in Settings->Apps and press Clear defaults button in Launch by default section.

Next time you try to open YouTube video you should be asked which app to use.

These instruction were verified on pure Jelly Bean (on Galaxy Nexus).