Youtube full screen doesn't work properly on yosemite
Just to confirm your observation on current version 38.01 of Firefox.

There are 2 types of full screen mode.

One is the Firefox it self in full screen mode. It will stay like that till you turn it off.

It does not look the same as the Video full screen. You can still see the Tabs. It is not correct that it will open the videos in full screen automatically, you still have to do that on the video it self.

The second mode is the Video full screen mode. It is controlled by the video player. In this mode the video takes truly the full screen and you do not see the tabs any more.

In full screen "video mode" all browser functions are disabled (not visible) thus you can not browse anyway.

Granted in your situation:

Firefox full screen -> Video full screen, when exiting video full screen also the Firefox it self will exit the full screen is what Firefox does currently.

That is (was) a known bug in Firefox.

One way it works better is to choose the Video player for YouTube. 

Switch from the HTML5 player to the Flash Player.