Witch and Youtube Fullscreen Windows
It turns out Chrome is apparently creating some special window called "Window". Witch sees that as the previous window and ⌘tab switched to it, not the actual video window. This takes me to the correct space for the video but it's the wrong window.

The solution is to tell Witch to ignore windows with the name "Window" by adding ,Window the the "Ignore windows with titles:" Behavior settings.