Commutative Hash Function for Password Challenge? I had an idea. Could anyone tell me if this is a thing, if they see any problems with it, and if they know of a commutative hash function that would make it work?

Suppose you had a pair of cryptographic hash functions, one of which could take an a key in addition to its regular input, and that these functions were commutative. So:

hash1(hash2(key, input)) = hash2(key, hash1(input))


I think this could be used to implement a very effective challenge-response password system:


Server stores storedHash = hash1(password)
User requests login
Server generates random key, sends to user. (Alternatively: User and hub use trust-less protocol to agree on random key.)
User returns sentHash = hash2(key, password)
Server compares hash1(sentHash) == hash2(key, storedHash)


The upside to this system is that even if the communications are intercepted, the users password remains safe, even for future use on the same server. Even if the user were tricked into trying to log in to a malicious fake server, the credentials they sent would be useless for impersonating them to the true server.

So... anyone see a problem with this? Anyone know a hash function that would make it work? (Or is there a reason any hash functions that's key-able and commutative would not be secure?)
If you are storing $h_1(password)$ and I break into your database, I will simply sort all the hashes I find. The one that appears most often will be $h_1($"123456"$)$, closely followed by $h_1($"password"$)$. In other words: you're storing unsalted hashes.

This is why you use special key derivation functions like scrypt and not plain cryptographic hashes on passwords.

If you are doing a man-in-the-middle attack, you generate your own key $k$ and send it to the user, who replies with $h_2(k, password)$. You know $k$ (this is still the case if you use a "trust-less protocol" to agree on $k$) so you can now try $h_2(k, x)$ for all $x$ in the 10 million most common passwords list at your leisure - even without breaking into the server.

If we're talking about key-based authentication, where collisions are unlikely to be a problem, there is an authentication protocol that uses a commutativity property a bit like you mentioned: Schnorr ZK proofs.