Disabling password saving for certificates
When the system prompts you to save your session in the following dialog:

Make sure that you do NOT check the Remember password in my keychain checkbox.  If you did check that, then you can always delete that saved password in the Keychain Access app, located in your Utilities folder:

Notice that even if you don't check the box, SSH does cache your password until a reboot.
Dave Dribin has an excellent breakdown of what's actually going on with OS X's SSH Keychain integration: http://www.dribin.org/dave/blog/archives/2007/11/28/ssh_agent_leopard/
Summary:

The real benefit, though, is the second checkbox: “Remember password in my keychain.” While it does store the passphrase in your keychain, it actually does more than that. It also adds the identity to your ssh-agent for you.
By adding your identity to the agent, you can now log right back into the same machine, without typing any passphrase. However, ssh does not prompt for you passphrase because it gets it from the agent, not your keychain. Remove your identity from the agent, and try again:

% ssh-add -D
All identities removed.


Locking your Keychain
The other thing to consider is just locking your keychain after a reboot or sleeping.  This will essentially give you the behavior that you're looking for as well.
You can do this from the Keychain Access utility, by right-clicking on the keychain that your private key's password is saved in, and clicking on Change Settings for Keychain '&lt;YOUR_KEYCHAIN_NAME&gt;'
The following dialog will appear:

Just select the one that best fits your compliance requirements.