Mask an Image Using Freestyle Lines
You may use the Set alpha node to mask the render result with the parts where the freestyles lines are by picking the freestyle layer alpha cannel.

Notice that I also switch the Base Thickness mode from Center to Inside (see the panel on the right) to be sure to always mask parts that are actually rendered geometry and not background.
With higher values of thickness you may have some problems with the cap in the angles. I would suggest to multiply the two incoming alpha to create a mask capable of cutting them away (Multiply Math node runs the same operation of a Logic AND):


Now that we've &quot;mastered&quot; the alpha channel in a simpe way, we can Invert it's apha channel (check the RGB box to invert its values as if it was an image from the colr channel) to get a stencil look to the image:

You can make the jiggled edges by using one (or a comobo of more) of the several modifiers available, for example a Noise for the Thinkness parameter:


If you need to use the freestyle layer color to mask the rendered object, you'll need to set it of a different color from total black, or you won't be able to get its information without accessing the alpha channel anyway.