Mathematics behind the creation of 3d face mask at runtime
Sorry for the naive answer, but why not consider reading the documentation of the library you have used?
http://dlib.net/
http://dlib.net/python/index.html#dlib.get_frontal_face_detector
http://dlib.net/python/index.html#dlib.shape_predictor
Quick googling also reveals numerous posts on how this library works and how it can be used
https://www.pyimagesearch.com/2019/12/16/training-a-custom-dlib-shape-predictor/
DLIB appears to be a machine learning / deep learning library, which means they would train a predictor like a deep neuronal network on some training data, and then use that network to place points on your images. The mathematics behind that is no different than any other neuronal network. Just find some course on machine learning / deep learning, and you will learn what you need.