How does UV unwrapping work? I would like to project the triangles of my mesh to a 2D surface. I would like to minimalize the distorion, avoid breaking connected polygons if possible, and also maximalize the used space on the 2D surface.

I tried to look for algorithms/papers about the topic, but I can only find articles about how to use UV unwrapping in 3D modelling softwares like Blender.
UV unwrapping is a difficult topic. They can be both combinatorial algorithms or variational methods but in general they're optimization based, i.e. you setup an optimization problem and you solve it using some numerical optimization solver. I'm just going to give you some names and some libraries you can use eventually. I'll give you few names (both classic and more recent) and you can have a look yourself. ABF (Angle Based Flattening), LSCM (Least Square Conformal Mapping), ARAP (As rigid as possible), BFF (Boundary first flattening), Variational Surface Cutting, OptCuts.

For ARAP and LSCM you can use libigl. Boundary first flattening was developed by Keenan Crane few years ago. Variational surface cutting is an improvement over BFF. And I recently discovered OptCuts. The code is available and you can run them yourself.

You need to read through the papers to understand how they works if you wanted to implement them from scratch (not going to be easy, parametrization is a difficult subject). But if you want to just to run them just clone the repos and try them yourself.
(libigl comes with tutorials, and two of these are about ARAP and LSCM). BFF, Variational Surface Cutting and OptCuts have a GUI based app so you can see the unwrapping and you can easily isolate the code if you need them in your own code.