Encrypt hash using hash of hash? I wonder what is wrong with this scheme for authentication.


Server sends random challenge, C 
Client returns (C, B(P)) encrypted in H(B(P)) where B(P) is high work factor salted hash of the user passphrase (e.g., bcrypt), and H() is a low work factor hash
Server has K = H(B(P)) in database, and can use it to decrypt (C,B(P)) and verify that H(B(P)) = K and that C is equal to the challenge


I wonder is B(P) adequately protected by encryption in key = H(B(P)) when sent over the network?  Or, other problems?

EDIT:  Let me clarify based on comments.  I obscured the main question by splitting the password hash function into 2 stages.  I think the answer to my question is the same if there is no B(P) and H() has high work factor.  So, client is submitting (C,passphrase) encrypted in H(passphrase).  Server knows H(passphrase), as in typical password authentication, so can decrypt.  Then can compute H(passphrase) to verify client knows passphrase.  It is like classic bcrypt based authentication, but using H(passphrase), rather than SSL, to secure the transport.  My concern is how to safely do encryption when the key is derived from the data being encrypted.

The reason for the 2 stage H(B(P)) in my original question is to minimize load on the server, which is the main aim of the protocol, to allow run of the mill hardware to separate wheat from chaff in face of gbps DoS attacks.

EDIT2:  Also, I left out how the server proves to the client that it also knows K, which is necessary to prevent active man in the middle.  In step 2 the client includes a random session key encrypted in K, which is used to compute a message authentication code on all subsequent messages, including the server return.
So your idea is that the client does the work in calculating the slow hash $B(P)$, and proves that to the server by using a hash as an encryption key. It is definitely not a standard way to do things, and has some problems.

In particular, any eavesdropped can trivially launch an offline dictionary attack on the password, so it is like your normal password database was constantly compromised (which should normally lead to "change your password" emails being sent). That means any passwords that are simple enough to be attacked are completely unprotected.

Instead, I would second the suggestion of using TLS. Even if you can come up with a strong password authentication system that works in the clear, you need authenticated encryption to prevent eavesdroppers from reading the authenticated content etc.

If TLS is simply out of the question, you could use SRP. It, too lets you offload all the hard work of using e.g. bcrypt to the client, by using as B(P) as the SRP password: the server only uses the verifier, meaning it does not need to do the calculation. Unlike your protocol, it prevents both passive attacks on the authentication (as well as MITM). However, you should use the agreed key to authenticate and encrypt the rest of the session, or an attacker can just jump in at that point, so complexity is not that much less than TLS.

(You mention patent concerns. IANAL, but Stanford has a royalty free patent grant for implementing RFC 2945, and anyway the patent may already have expired.)