Context-based gap-fill face posture-mapper GAN
I believe you may want to use a Sum Product Network for this task. SPNs are the state-of-the-art approach for face completion, and there are several more recent papers on this topic since the original above.

Importantly, the SPN paper also covers other approaches that work well for this task. If lower-resolution results are acceptable for your task, PCA with 100 or more components works surprisingly well. If your dataset is very large, a nearest-neighbor approach can work decently too.