Mac OS X Mavericks keeps jumping to the first desktop
It is Google Drive, it's causing Finder to crash and restart, which is why it keeps jumping back to the first desktop. 

It's related to the "Show file sync status icons and right click menu" setting. If you turn this setting off in the Google Drive preferences, the problem should go away.