How to design a secure challange-response authentication protocol? I am implementing a custom challenge-response protocol to authenticate a client to a server. Mutual authentication is not required, i.e. the server doesn't need to be authenticated to the client.

The protocol works over TCP. Client and server are in a LAN (not over the internet). There is only one server and only one client. The authentication is designed as follows:

$S \rightarrow C : N$

$C \rightarrow S : \{N\}_K$

Where the server $S$ creates a 128-bit random number and sends it as nonce $N$ to the client $C$. The client encrypts the nonce with AES-192 with the pre-shared key $K$ and sends it as response to the server. Then the server decrypts the message and checks the nonce.

Questions:


Is this protocol design secure?
What would be the advantage to use HMAC (or any other MAC) instead of AES encryption?
Did I miss something?
Is this protocol design secure?


See below. 


  What would be the advantage to use HMAC (or any other MAC) instead of AES encryption?


HMAC was originally proposed as a construct that turns a Merkle-Damgaard hash function based on compression functions built from block ciphers, into a message authentication code. Although there's no decryption in HMAC, you can actually easily replace your AES decryption-based authentication with comparing HMAC-signed tags. 


  Did I miss something?


What you're doing is essentially an authenticated security transport without confidentiality. 

What you're missing is that you didn't authenticate the exchange following the initial authentication - this allows for connection hijacking, arbitrary injection of data packets, and more. 

Therefore, you should use HMAC+[hash] or CMAC+[block cipher] to authenticate all of your messages exchanged, and also keep a counter as a state to detect duplicate packets and message replay attackes.