Is OpenSSL HMAC vulnerable to length extesion attacks when using MD5 as hash algorithm?
HMAC-MD5 is not vulnerable to length-extension attacks, period.

If that's what your code computes, then your code is not vulnerable to length-extension attacks.  That said:


Length-extension attacks are a part of the protocol, not an implementation of the protocol.  It doesn't matter whether you use OpenSSL or something else to implement the same protocol.

You should have a concise description of your protocol and generate test vectors, e.g. with Python or Go, to confirm that your C code is computing the cryptography you intend in the protocol.  You should also have a clear statement in a design document of what your protocol accomplishes, without reference to the cryptography it uses; and a clear statement in a separate section of the design document of what security goals your protocol aims for.
Your code has another problem: it has a side channel leak that could be exploited to forge messages, because you're comparing secrets with memcmp.  In OpenSSL, you should use CRYPTO_memcmp instead.