Converting Container Problems
First reason:
Videos / audios in AVI or MKV containers may be encoded by various codecs, and not all them are supported by MP4 containters.

Second reason:
Containers are more than simple boxes for videos, audios, subtitles, etc. with some not so important metadata as author or date/time. They have a lot of unevitable valuable information, e. g. the synchronization between audio and video is possible only due to it.

So there is not a reliable algorithm to convert any AVI -> MP4 or any MKV -> MP4 (and, generally speaking, from one type of container to a different one) without changing the streams (audios, videos, etc.) contained in them - the result may be similar to the one you described.