How to choose keys in keychain access?
Put your per-server SSH-configuration into ~/.ssh/config like this:

Host useKeyOnly.net
user sshuser 
Identityfile ~/.ssh/mykey1
IdentitiesOnly yes

Host usePasswordOnly.org
PubkeyAuthentication no
Port 53221
user root