Masking out non-linear shapes on canvas
I'm not aware of painters tape being sold in super-wide rolls.  But you could replicate that by overlapping as many strips as needed to contain the shape inside a wide border.  Do that on an easy-release surface like wax paper or used backer paper from label sheets (use as much as you need to protect the adhesive).  Trace or draw the desired shape onto the tape and cut it out.  Then peel the tape from the backing and position it on the surface you want to paint.

Another approach would be to create a template.  There are lots of suitable materials you could use.  Chipboard is available in large sheets.  You might even be able to get a sufficiently large piece by opening up something like a large cereal box.  Some materials will hold up better than others if you want to reuse it many times. 

Along the lines of your vinyl idea, the vinyl used for laser-cutting lettering and shapes for signage is typically thin, and backed with an aggressive adhesive, although you could leave the liner attached.  You might be able to find heavier-weight, stiffer vinyl sheets at an art supply store.  

For small shapes, there are vinyl folders that are long lasting versions of manila folders; you could cut off the fold to make two flat pieces that would be good, long-lasting template material. If you don't have access to a laser cutter, you could cut it by hand with a hobby knife or hot knife. You might be able to use a French curve as a knife guide (though not with a hot knife).  

For larger sizes, you could use something like vinyl shelf liner, which is heavy enough to not stretch, or vinyl wall paper.  Tyvek, which is available in rolls, might be suitable for single use; it is likely to wrinkle, though, with the first use (but it's cheap and you could make more, cutting the shape out with scissors).   

The problem is in sealing the template against the medium so paint doesn't wick under it.  You need a temporary adhesive that doesn't leave a residue and isn't so aggressive that it can pull off what may already be on the medium.  If the medium is a textured surface, you also need something thick enough to fill the gaps.  

Rolling a thin bead of tacky putty (the kind used to stick things to a wall that doesn't leave any residue), would be a good material for that.  Line the back of the template right at the cutout edge.  If it is a large template made of heavyish material and the medium won't be horizontal when you paint, or material that wants to roll up, use clips of some kind to secure the template in position (or painters tape around the edge).  Then press the cutout area into position to adhere the putty.

Let me add a totally different approach.  There are temporary coatings that can be sprayed or applied with a roller.  They dry to a protective, clingy plastic film that can be peeled off without removing paint or other surface materials.  Typical use is to protect finishes or materials during storage or construction.  There are lots of similar products (Google "spray-on temporary coating").  Here's one from an Australian company, but similar products are available all over: https://www.shieldnpeel.com.au/.

To use it as a template, cut out a positive of the shape and position it on the medium.  Spray or roll a coating of the material over at least the edges of the shape, and wide enough to protect the area around the shape.  Extend it to an edge for easy access for peeling.  While it is still wet, remove the positive, so the coating is the template.  Peel the film off when you're done.