All Desktops swiping brings the app front
As long as another app that is set to None or This Desktop in the Desktop you are swiping from is the Active Window, and you did not make the app set for All Desktops active during the session on the previous desktop, then the Window that tracks to the All Desktops will remain in the background without coming forward and then going back. 

This behavior only happens when the All Desktops App is the Active Window when you swipe to the new desktop. I assume the reason for it is that the OS X designers felt that swiping and hiding the open windows on the new Desktop could cause confusion, so they added this animation to emphasize that the Active App from the previous desktop is deprioritized and that there are open Apps on the new Desktop.