What algorithm is used for "beauty" triangulation
After a quick look at the source code, I believe it tries to find a triangulation that leads to the best angles for each triangle (ie biggest minimum angle and smallest maximum angle).
For a quad, there are only two possible splits so it tries both.
For a polygon with more sides, it does some sort of greedy iterative search of an initial triangulation and edge rotations.
Possibilities where triangles that are flipped relative to the initial polygon are discarded.
Relevant code:

bmo_triangulate_exec in https://github.com/blender/blender/blob/master/source/blender/bmesh/operators/bmo_triangulate.c
BM_mesh_triangulate in
https://github.com/blender/blender/blob/master/source/blender/bmesh/tools/bmesh_triangulate.c
BM_face_triangulate in
https://github.com/blender/blender/blob/master/source/blender/bmesh/intern/bmesh_polygon.c
BM_verts_calc_rotate_beauty in
https://github.com/blender/blender/blob/master/source/blender/bmesh/tools/bmesh_beautify.c
BLI_polyfill_beautify in
https://github.com/blender/blender/blob/master/source/blender/blenlib/intern/polyfill_2d_beautify.c