How to make OS X not remember ssh key passphrase
While reading GitHub documentation about adding ssh key to the ssh agent, I thought you might be able to prevent loading keys in the agent by modifying the ~/.ssh/config file like so:

Host *
 AddKeysToAgent no
 UseKeychain no
 IdentityFile ~/.ssh/id_rsa