Using the builtin dialog under OSX to unlock an SSH key
I think the dialog is because it is adding the ssh key to the keychain (as well as|instead of) to the .ssh/ files.

You should be able to invoke ssh-add -K and get the same results.