Is there a difference between asking for a signature and asking for a decryption in challenge-response authentication?
As long as properly implemented using secure algorithms, there is no real security difference. In both cases the protocol is secure as long as the underlying signature or encryption algorithm is.

However, one difference is the random number used:


In scheme A the numbers must be unique. If the server ever reuses a number, then an adversary could replay a previous signature they have observed.
In scheme B the numbers have to be unpredictable, or an adversary can authenticate just by guessing correctly, but as long as the encryption algorithm is semantically secure, repeats are not a problem.


You need larger random numbers in the first scheme than the second, due to the uniqueness requirement.



As mentioned in the other answers, this is not secure against man in the middle attacks. It could be run in a channel that prevents those. For example, in a typical TLS situation the client would have authenticated the server already, man-in-the-middle attacks would be ruled out, and the server could use either of these to authenticate the client.