Does this 3D rendering method exist? How is it called? Is there any C++ algorithm for it? I've been thinking about making my own 3D rendering engine in C++, but I don't know much about math that's required to do it. I'm not even sure if it's possible or if it even exists or how it's called.

It goes like this. Let's imagine a 3D body that has 6 vertices and 8 faces. It's like a pyramid on top of an upside-down pyramid. Each face is a triangle, but let's imagine that every line between the triangle points is a three-dimensional bezier curve. That would make each face look like a tent. If we do that, then we get a sphere. Now, imagine if instead of rendering that sphere by subdividing the tents into triangles we render the tents exactly how they are and imagine if we don't render raster graphics, but rather vector graphics in PostScript/Flash/ActionScript/GhostScript format in ASCII that can be opened by Notepad and rendered as a semi-transparent PNG image at any resolution without losing quality.

When rendering 3D triangles, matrix operations are used if I'm not mistaken. There's a matrix with some values and then a determinant is calculated. But these values are constants. What if the values depend on a formula of the bezier curve that's in 3D space? How can we stuff the formula into the matrix? And a better question; how can we stuff that into an algorithm or into OpenGL or into a language that a graphics card can process instead of the CPU? Does such a formula, matrix or an algorithm exist?

Now, I'm not thinking about rendering simple spheres, but rendering something like this: https://www.blendswap.com/blends/view/24878
I'm thinking about making a lightweight renderer in assembly for an ESP32 SoC or some other Arduino microcontroller/processor/SoC which would render all that onto an LCD or a PAL/NTSC TV via a composite AV jack, but that's another topic.

My question is whether or not such a rendering method exists, how it's named, is it free to use, is it patented and is there any permissive-licensed C++ code for it (MIT, zlib/png, Apache, BSD). Please let me know.
Yes, such rendering exists. It is called a parametric surface and there are several methods you can render such entities. In this case you are talking of bezier patches.

Turn the object into enough triangles. The triangilation is called tesselation or meshing depending on what literature you read.
There are lots of applications that do this allready. Most cad applications work this way and many graphics apps like blender can work
this way.

raytrace or raymarch the surface in a shader.

Microdice the surface

etc.


The reason this is not often done is that the on line tesselation is for most intent and purposes just a hindrance to asset developpers. You get the same with subdivision schemes and its just easier. The primary reason to drag the bezier (or nurbs, or t-splines...) is that unless you need to find better intersection of underlying primitives then it is often just a waste of resources. Although subdividing your mesh  via a subdivision surface is allready more useful