Algorithm to stack astronomical images
This page about a commercial product goes into some detail about their algorithm.  It does the triangle matching you describe, with something like simulated annealing to get a more optimal solution.

The accepted answer to this closely related question recommends Hugin panorama software; it's open source, so you should be able to glean the algorithms used.