ssh key persists even after I delete the private key from ~/.ssh -- why?
Welcome to the wonderful world of OS X Keychain and ssh-agent! From the man page:


  ssh-agent is a program to hold private keys used for public key authentication (RSA, DSA, ECDSA).  The
       idea is that ssh-agent is started in the beginning of an X-session or a login session, and all other
       windows or programs are started as clients to the ssh-agent program.  Through use of environment vari-ables variables
       ables the agent can be located and automatically used for authentication when logging in to other
       machines using ssh(1).


Your private key was cached in the agent and is being fetched from there to authenticate your session. OS X, via Keychain, does automatic ssh-agent caching for you because it's "user friendly". It can be a little alarming, but it's mostly a handy thing that this happens.

The default time-to-live for cached keys is forever. That's probably not a great default. It'll flush the cached keys on a logout/login-cycle or a reboot. Or you can force a flush by killing the current agent using:

ssh-add -D


You can list all credentials cached by the agent by running:

ssh-add -l


This will do the same but show the full public keys:

ssh-add -L