Second password for SSH-only access
I think the real solution is to create a non-admin account which can be accessed via ssh (presumably the account you want to use via physical access will have sudo privileges). The ssh account should not really have anything or be able to do anything other than let a user log in. You can then sufrom that user to the actual account you want to use, which will give you an additional layer of security and better logging regarding who is accessing the system (or trying to)