Use OpenSSH agent instead of the OSX agent
ssh-agent, the system builtin one, is run as a system daemon started by /System/Library/LaunchAgents/com.openssh.ssh-agent.plist. Cause the service plist is located in /System/Library/LaunchAgents, it mean it's impossible to disable it unless you break SIP, which is not recommended.

The macOS version ssh-agent is built with additional KeyChain support compared with the standard one. The best solution is to use the keychain.

First, uninstall the homebrew-installed SSH, brew uninstall openssh.

Secondly, configure the SSH to use password from Keychain.

Host *
AddKeysToAgent yes
# use the password stored in keychain
UseKeychain yes
IdentityFile ~/.ssh/id_rsa


Thirdly, add the key and password into Keychain.

ssh-add -K ~/.ssh/id_rsa


In this solution, we keep the password into Keychain and configure OpenSSH to load password for the key from Keychain.

References


OpenSSH updates in macOS 10.12.2