ssh-add every time I want to use SSH to login to my remote server
I know it does not answer your question directly, but have you tried using a config file to map you keys to servers?

in .ssh create a file called config
In the file put your server entries and map the key

Host server1
HostName  10.10.10.1
User myUser
IdentityFile ~/.ssh/id_rsa


then you can just run ssh server1