What kind of video encoder can do this? Blend multiple video frames together live
While there are video encoders that do this kind of thing, most commonly this is the work of a video switcher or mixer.  A video switcher is a device used for mixing live video from multiple sources.  The bad news is they generally aren't cheap.  An HD capable switcher is typically in the $6000+ range with the absolute cheapest one I know of  being the Black Magic Design Studio ATEM which is $1000 and requires use of a software interface to control it.

The switcher then would output the final composite video that would go in to an encoder to be sent out, or in some newer cases, may even provide encoding functionality itself (such as the previously mentioned ATEM Studio).

Alternately, you probably can find software that will attempt to do the scaling and encoding operation on the fly, but it is going to require quite a beefy computer to pull it off and you'd likely end up spending more than the $1000 for the ATEM Studio to build a powerful enough system to handle it with any high level of quality.