Why do vertices move when trying to box select? I accidentally hit some buttons and got stuck in some kind of different edit mode. I can't box select anymore, when I try to, the selected vertices will move. I can still select by single-clicking.

I don't really want to reload because I haven't saved the upper torso yet. It's not that much work but I rather don't do it again.

Can someone tell me the magic hotkey to get out (ctrl + z did not work)? And what this mode is?
You've pressed W. This is mapped to the selection tool and will cycle through the different selection methods if the select tool is already highlighted.

Either press W until you're back to box select, or click and hold the Selection button in the toolbar below.