FFmpeg - Synchronize external audio file
One way to achieve this might be to create an .MP4 with your audio and an all-black image, which should encode to something not much larger than the audio file but still have the same internal structure as the full video files.