ssh-askpass on Mac OS 10.9/Mavericks broken?
Problem solved! When in doubt, read the source...

ssh-agent won't call ssh-askpass unless the DISPLAY env variable is set.
Most likely it has been working on earlier installations, since I had X11 installed. Not on this box (yet).

Workaround:

Patch the ssh-agent LaunchAgents file:

--- org.openbsd.ssh-agent.plist.orig    2014-05-22 09:55:53.000000000 +0200
+++ /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist    2014-05-22 09:56:31.000000000 +0200
@@ -9,6 +9,11 @@
        &lt;string&gt;/usr/bin/ssh-agent&lt;/string&gt;
        &lt;string&gt;-l&lt;/string&gt;
    &lt;/array&gt;
+   &lt;key&gt;EnvironmentVariables&lt;/key&gt;
+   &lt;dict&gt;
+       &lt;key&gt;DISPLAY&lt;/key&gt;
+       &lt;string&gt;:0&lt;/string&gt;
+   &lt;/dict&gt;
    &lt;key&gt;ServiceIPC&lt;/key&gt;
    &lt;true/&gt;
    &lt;key&gt;Sockets&lt;/key&gt;


Then restart your computer. 

Or reload it by force, however SSH_AUTH_SOCK will be invalid in all new terminal sessions:

launchctl unload /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist                        
launchctl load /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist
export SSH_AUTH_SOCK=$(launchctl getenv SSH_AUTH_SOCK)                 
ssh-add -c ....


Problem solved!

Note: the ssh-agent plist file will probably be overwritten on upgrade... Might be cleaner ways to ensure DISPLAY is set.