In making a mask, how do I remove the outer area and lay the inner area atop video?
Compositing

Solution with compositing like you suggested with the screenshot.
Using the mask as an factor input mentioned in the comments above.
(I just use the colors as a placeholder, you can use renders, movies, images...)


Video Sequence Editor (VSE)

Another solution would be to use the Video Sequence Editor (VSE). You can add both the render or what you want to cut out and the video. The video under the 'overlay'. On the strip you want to overlay you can add a mask modifier. Don't forget to change the blend type of the strip above to 'Alpha Over' or 'Over Drop' otherwise it will replace the image below.



Inverting the mask

To invert the mask, go to the mask editor (image editor and then select the mask view). In the right panel (N-Panel) there is the option for the mask layer. On the right of the 'Opacity' input there is the invert switch (white and black rectangle). By clicking it you can invert the mask.

(Tip: If you don't see a change. In the composition may enable auto render. In the VSE press 'Refresh Sequencer'.