How to prevent html5 videos from using built in mac fullscreen
I'm afraid that's not possible.

Since all major browsers (Chrome, Firefox, Opera and Safari) all use the native Fullscreen API, you're stuck with the slow animation...

I researched wether there were any work arounds or tricks but there doesn't seem to be. You're not alone in being annoyed.

Only thing you can do is to use Flash when available, or, upgrade to Mavericks where the animation is minimal and quite a bit faster. 10.9 happens to be free so the entry barrier isn't that high.