SSH, passphrases, keychains, Could not open a connection to your authentication agent
The ssh-add -K  command add the key to the Apple's Keychain, but there is a SSH specific keychain, unrelated do the Apple's one.

It runs as a deamon and interacts only with SSH.

I don't know if this will help you, but I have strong feeling that it will.

Keychain is available as a homebrew package so if you get nomebrew available at your system to install it just type brew install keychan. After that a man keychain will help a lot.