Allocating matrices for in-place modification
Answering some of your questions...


  Is passing arrays to functions memory safe?


No, not really, but the C language offers no other approach, so you just have to be extra careful, since neither the compiler nor the runtime will alert you if you try to access invalid or out-of-bounds memory.


  I am allocating matrices on stack and passing them to function to modify in place. Is this a standard practice?


It can be, yes. If you prefer to have them modified in place, it is fine. Some libraries prefer to have separate input and output parameters, and that tends to produce cleaner code, but sometimes it's simpler to just modify in place. E.g.: It would be annoying to have, for instance, a "make identity" function in a matrix library take two parameters, like: matrix_identity(const mat* in, mat* out). Much more convenient to just write:

matrix_identity(&amp;matrix);


However, for arithmetical operations it would be more natural to have inputs and outputs:

mat a = { ... };
mat b = { ... };
mat result;
matrix_multiply(&amp;result, &amp;a, &amp;b); // result = a * b


As to allocating on the stack, leave that up to the user. Your code should only be concerned with pointers to objects, how that's allocated is up to the caller. Leave the option for both, since stack variables are cheap, but sometimes one might have to use the global heap to extend lifetime.



Other details about your code:


The error codes you're returning in each function are irrelevant, since every function always returns the same MT_OK, which means they could very well be void functions. You don't have to make all functions return an error code for the sake of uniformity. Save the error codes for functions that can actually fail for some reason.
Since C doesn't have any construct for scoping names (like modules/namespaces), it is common practice to prefix your function names with a pseudo namespace, to reduce the chance of collisions. You could maybe prefix your function names with mat4_, e.g.: mat4_ translate(), mat4_identity(), mat4_mult_vec4() etcetera.
Don't redefine PI risking using less precision than the actual supported. Most math.h implementations will have M_PI defined, which you can use. You may still provide a fallback if necessary, but default to M_PI:

#ifndef M_PI
  #define M_PI 3.14159265f
#endif // M_PI

Which matrix layout are you using? Column major, I suspect? State that very clearly somewhere, like in a documentation comment in the header file, to leave no room for confusion.