Enable ssh-key-only login to my Mac for 1 user
To disable password authentication you need to edit /etc/ssh/sshd_config. In Terminal:
sudo vim /etc/ssh/sshd_config

(or replace with your favourite editor)
Look for the line:
#ChallengeResponseAuthentication yes

And change it to:
ChallengeResponseAuthentication no

That is, remove the # at the beginning, and change yes to no.
All subsequent ssh logins will now REQUIRE you to use an ssh key and will not prompt for a password. Ensure your ssh key works before doing this if you have no local access! There's no need to restart the ssh daemon or anything because it is started on demand for each new incoming connection. Note that you may also see a PasswordAuthentication line, but that defaults to no already.
If your ssh keys aren't working then that's a separate issue really, but this answers the question as it is posed now :)