Stack videos with ffmpeg without padding
The stack filters don't insert any padding. Your videos probably have empty space in them. You'll need the crop filter to remove the dead space.

The stack filters require the video edge along the join axis to have the same dimensions i.e. height for hstack and width for vstack. So, don't crop those aspects.