Goal
====

I want to record/generate a video in browser (Chrome especially) with a custom video (e.g. .mp4, .webm) and camera recording side-by-side.

```
--------------------------------------------------
|                        |                       |
|  Some Custom Video     |       My Camera       |
|                        |                       |
--------------------------------------------------

```

What is working
===============

I can use `MediaRecorder` to record my camera, and play the recording side-by-side with my video, and download the recorded video as a webm.

Challenge
=========

I'm facing difficulty merging the video and camera recording into a single video file side-by-side.

My investigation
================

### MultiStreamMixer

I first looked into MultiStreamMixer and built a demo with it (See [codepen](https://codepen.io/lzl124631x/pen/oNbOLKV)).

[![enter image description here](https://i.stack.imgur.com/ZypRM.png)](https://i.stack.imgur.com/ZypRM.png)

The issue with it is that it stretches the video content to fit them in the same size. I can specify different width/height for those two streams but it doesn't work as expected. My camera got cropped.

[![enter image description here](https://i.stack.imgur.com/IkM5F.png)](https://i.stack.imgur.com/IkM5F.png)

### Custom Mixer

I took a look at the source code of MultiStreamMixer and found the issue was because of its simple layout logic. So I took its source code as a reference and build my custom mixer. See [codepen](https://codepen.io/lzl124631x/pen/jOWoWaL).

The way it works:

* We first render the streams one by one to an offscreen canvas.
* Capture the stream from the canvas as the output video stream
* Audio stream is generated separately using `AudioContext`, `createMediaStreamSource`, `createMediaStreamDestination` etc.
* Merge the audio and video streams and output as a single stream.
* Use `MediaRecorder` to record the mixed stream.

It adds black margins to video/camera and won't stretch the videos.

[![enter image description here](https://i.stack.imgur.com/sbagi.png)](https://i.stack.imgur.com/sbagi.png)

However, I found the recording is very blurry if you wave your hand in front of your camera while recording.

[![enter image description here](https://i.stack.imgur.com/6HTNa.png)](https://i.stack.imgur.com/6HTNa.png)

Initially I thought it was because I didn't set some setting correctly to the canvas. But later I found that even the [my MultiStreamMixer demo](https://codepen.io/lzl124631x/pen/oNbOLKV) or [the WebRTC demo](https://webrtc.github.io/samples/src/content/capture/canvas-record/) (You can't see the text on the teapot clearly in the recording) generates blurry video with canvas.

I'm asking in webrtc group to see if I can get around this issue. Meanwhile I looked into `ffmpeg.js`

### ffmpeg.js

I think this would "work" but [the file is too large](https://rawgit.com/bgrins/videoconverter.js/master/build/ffmpeg.js). It's impratical to let the customer wait for this 23MB JS file to be downloaded.

Other ways that I haven't tried
===============================

The above are my investigations thus far.

Another idea is to play the video and recorded video side-by-side and use screen recording API to record the merged version. ([Example](https://dev.to/sebastianstamm/screen-recording-in-10-lines-of-vanilla-js-3bo8)). But this would require the customer to wait for the same amount of time as the initial recording to get the screen/tab recorded.

Uploading the video to server and doing the work in server would be my last resort.