Hotkeys don't work in zsh when $TERM is set to screen-256color
Seems like I finally found an answer! Or rather a workaround, since I'm still not sure what exactly is going wrong.

Explicitly configuring the key binds for word navigation fixed the issue for me:

bindkey "^[f" forward-word
bindkey "^[b" backward-word


Maybe zsh is using different keybindings based on $TERM.