Is there a Blender alternative to Chroma Key rendering in OpenGL?
To clarify, this is offered as an alternative to Chroma Keying and as far as I'm aware, will work only when rendering in OpenGL.

"Inviso" is a term I've adopted to describe an effect of invisibility (in the 3D viewport and cameras) when an object's transparency is set to just above 0. (typically .008 or even less)

When rendered using OpenGL, the object is not only invisible, but so is anything (or portion thereof) that happens to get behind it.

The effect can be extremely handy in many cases that would otherwise require Chroma Keying and it's accompanying node setup, plus whatever extra rendering is needed to apply the resultant keyed alpha layer.

Chroma key in effect, makes an alpha layered 'hole' in a video clip.  

Inviso can do this also but it doesn't have to - it's selective!

i.e. Objects behind the mask can be selectively rendered invisible so that some disappear and others won't, including a backdrop (textured) plane.

Even better - the effect is achieved 'live' within Blender's 3D window.  Rendering using -= OpenGL =-, outputs your scene and the masking effect all to video in the one go!

I've not tried this but see little reason why animated shapekeys can't also be used on 'Invisos' to change the shape of a mask 'live' during an animation sequence, perhaps to match a changing viepoint or to 'hug mask' an object, such as this tree trunk below.  



The above was achieved on a still image by adding vertices to the mask where needed and deforming it's shape to match the tree trunk's.  The smaller cube itself is in fact, still sitting in front of the photo, the Inviso mask in front of that again to 'cut' the tree trunk's shape out of the cube.

The post/answer this was used in is here -

Realistic rendering of mesh together with image?  (Scroll down a fair way)

Note - Surfaces so shape-keyed would preferrably not want to be buckled to avoid shading artifacts.  On a flat plane, that wouldn't be difficult to enforce.

Two more examples where this method of masking was used (Scrolled text and opening a scrolled parchment) -

how do i achieve this simple text animation?

...the gap between the bottom of the screen and the black box with the text is masked using the same method.

Below - is what is seen when rendering in OpenGL where the Inviso plane (ordinary 2D plane) is not set to total transparency and consequently still taken as 'solid' by either Blender's code or the hardware.  

Selective Inviso effect -  



The plane's transparency has been set to 0.008 (or close to it) and it's transparency specular set to 0, it's ordinary shading specular also at 0.

The cube is affected whereas Suzanne(?) our monkey and the ground plane are not. (See further below as to how this is achieved)



Tick or untick it's transparency box in it's properties panel (orange cube icon at right) under the "Display" tab .

When this is done, often by default the plane will then render any object (or part thereof) that's behind it, invisible.  

Note -

If animating the Inviso mask, leave that until all objects involved in the masking effect are setup and are being rendered as desired.

If you do Inviso an object after the mask is animated, you'll need to unlink the mask and object's animation data after parenting! (in the layout panel at top right)

-=======-

The criteria for making a mask work reliably can depend on how the mask's  "Properties (cube icon)-->Display-->Transparency" box is ticked. (at right)

Tick/untick this box to make it work properly. 

Don't confuse the box with it's Transparency tab found under it's "Material" settings.

When viewing or rendering, make sure to -= DESELECT =- everything first! (press A)

-=================-

To Be or Not to Be -(Selective invisibility) -

If an object isn't by default, disappearing when behind the mask, simply "Parent" it to the mask. (CTL-P)  Then unparent it again. (ALT_P)

--

EDIT:  22nd April 2017-

To immunize an object from the effect altogether, you have two choices -

The first is simply reverse parent.  i.e. parent the mask to the object that's disappearing, then unparent again.  Goto the "Outliner' strips and remove any strip that doesn't directly belong there.

-= Make sure to read the beginning of the EDIT (below) '15/3/2017' =-

The second and more difficult method is -

Join it to the mask, then part it again.  

A word of caution here.  Join the object to the mask. (select the object first, the mask last before pressing CTL-J)

When separating again, select the mask, press "Z" for wireframe.

Goto Edit mode, select vertice mode, box select all vertices in the object, (not the Inviso plane) 

Press P to "Part by Selection".

The object should be seen normally again when you press A.
You may have to tick/untick that Transparency box in the "Display" tab.

Once separated, it's a good idea to reset the mask &amp; object's origin back to their respective geometry.

-======-

Edit: 15/3/2017

A trap for young players with Inviso's are the properties strips in the Outliner.  (top right) 

Any mask or objects that have been parented/unparented, joined or separated, may inherit each other's outliner strips.

This is not a problem when only one mask is being used.  It can become a nuisance however when two or more masks are employed. Objects that were rendered invisible can become visible again to the other mask(s)

The solution is two-fold -

Never create the second mask independantly.  Instead, duplicate the original mask and use the duplicate.

The other factor is to purge both the masks and associated objects' "Outliner" entries of anything foreign. (strips that don't belong)  Right click on each and select "Unlink".  It's a good idea to make a save of the project before embarking on this.

The only kink that may remain might be the mask's "Transparency" box needing a tick, or perhaps to be unticked.  This is under it's little cube icon at right-->Display-->Transparency.

With both methods -

A reminder to deselect -= EVERYTHING =- before viewing !

I've recently used 5 masks in a project with multiple overlaps, all without problems.

-============-

Video Output -

Rendered video formats inclined towards colour banding, such as mp4, gif anim etc, can reveal a mask very very faintly.  

This might be the result of a specular setting not being at zero, or a particular lighting angle.  Both are easily fixed as a rule.
Adjusting the mask's Diffuse Intensity &amp; colour slider is another way of eliminating a difficult 'ghost'. 

A reminder - Render in OpenGL. (Top left of 3D window under - "Render")

If with some future version of Blender, this effect ceases to be available, simply revert to the last version where it was.
(in my case that's 2.77a) 
Make Inviso clips on that exclusively.  

I for one hope the effect will stay there.  It can save significant time and effort.