Is there a difference between asking for a signature and asking for a decryption in challenge-response authentication? Consider the following two authentication schemes. In both cases the server knows a public key of the client (for a signature system in the first and an encryption system in the second case).

Scheme A

Server presents a randomly generated plaintext to the client, and asks the client to sign the message. The server can verify the signature using the public key.

Scheme B

Server randomly generates a plaintext, encrypts the message using the client's public key, and asks the client to decrypt and send back the plaintext. The server can verify that the plaintext matches the one it generated.



The second scenario feels less secure than the first scenario, but I cannot quite put my finger on why.

Is there any effective difference between the two authentication schemes?
As long as properly implemented using secure algorithms, there is no real security difference. In both cases the protocol is secure as long as the underlying signature or encryption algorithm is.

However, one difference is the random number used:


In scheme A the numbers must be unique. If the server ever reuses a number, then an adversary could replay a previous signature they have observed.
In scheme B the numbers have to be unpredictable, or an adversary can authenticate just by guessing correctly, but as long as the encryption algorithm is semantically secure, repeats are not a problem.


You need larger random numbers in the first scheme than the second, due to the uniqueness requirement.



As mentioned in the other answers, this is not secure against man in the middle attacks. It could be run in a channel that prevents those. For example, in a typical TLS situation the client would have authenticated the server already, man-in-the-middle attacks would be ruled out, and the server could use either of these to authenticate the client.