How can I use an ssh key with a passphrase but not have to type the passphrase every time when working in a remote shell?
The solution is called ssh-agent. It can remember your passphrase for the session. You run the agent with

eval `ssh-agent`


(might be even running in your session already, so check before using ssh-add -l) and add the key to the agent using:

ssh-add /path/to/your-private-key


it will prompt you for a passphrase. The further connections using this key will not prompt you for the passphrase.