Why does FFmpeg's stream_loop option add a pause?
This is a known bug with stream_loop. See https://trac.ffmpeg.org/ticket/5719

Wrapping to MKV or TS and then looping may work. Re-encoding your video as intraframe is the safest option.