Is OpenSSL HMAC vulnerable to length extesion attacks when using MD5 as hash algorithm? Given that a valid hash for a specific payload is known, would some C code like this be vulnerable to a length extension attack?

#include &lt;openssl/hmac.h&gt;
...
unsigned char *calculated_hash = HMAC(EVP_md5(), &amp;key,sizeof(key) - 1, payload, payload_len, NULL, NULL);
...
    if(memcmp(calculated_hash, received_hash, 16) == 0){
        dprintf(client_fd, "Payload Valid\n");
        exit(0);
    } else {
        dprintf(client_fd, "Payload Invalid\n");
        exit(1);
    }
HMAC-MD5 is not vulnerable to length-extension attacks, period.

If that's what your code computes, then your code is not vulnerable to length-extension attacks.  That said:


Length-extension attacks are a part of the protocol, not an implementation of the protocol.  It doesn't matter whether you use OpenSSL or something else to implement the same protocol.

You should have a concise description of your protocol and generate test vectors, e.g. with Python or Go, to confirm that your C code is computing the cryptography you intend in the protocol.  You should also have a clear statement in a design document of what your protocol accomplishes, without reference to the cryptography it uses; and a clear statement in a separate section of the design document of what security goals your protocol aims for.
Your code has another problem: it has a side channel leak that could be exploited to forge messages, because you're comparing secrets with memcmp.  In OpenSSL, you should use CRYPTO_memcmp instead.