Image distortion and alignment on 3D mesh / pointcloud Currently I have a fairly accurate (50 µm) and large dataset from a 3D scanner and a set of high resolution pictures. Sadly the 3D scanner (a Nikon/Metris arm-based CMM that was hanging around the lab) doesn't record colour data, but this would be exactly what I need to extract the actual features I'm interested in. I started out by correcting the lens distortion for the camera, and also accounted for the non-linear distortion. What prevents me from doing this manually in software like Meshlab or Geomagic is that I have an extremely large set of objects to scan, which makes it highly unrealistic to do it by hand.

I took the liberty to add a series of reference points which are visible to both the 3D scanner and conventional camera. I am able to detect these features on both the point cloud and images automatically. Sadly I must admit I underestimated a big part of this task: I now wish to automatically align the image versus the point cloud based on these reference points and project the image onto the point cloud or mesh. The reference points from the point cloud are in a XYZ format, while those from the image are 2D coordinates in the plane. They are already matched, so that presumably - according to a fellow Ph.D. student - difficult part is already done. And while I can think of a few ways to align, distort, and project the images this would most likely be a large investment of time from my part. I was wondering if any pre-existing libraries for this exist or at least if there are papers which present a method to do so. Based on the fact that software like Geomagic is able to do it I'd like to avoid reinventing the wheel. However I must humbly admit I can't seem to find them as I am probably using the wrong terminology.
If you're only left with matching 3D points to their corresponding 2D parts, this is a fairly task and called PnP pose estimation. OpenCV has a lot of options to solve this problem. Here is one tutorial, which you could maybe benefit.