Matching points between 2 polygons
This is a large topic, often called "geometric shape matching." Here is one survey that can lead to many specific algorithms:


  H. Alt and L. J. Guibas. Discrete geometric shapes: Matching, interpolation, and approximation. In J.-R. Sack and J. Urrutia, editors, Handbook of Computational Geometry, pages 121–153. Elsevier Science Publishers B.V. North-Holland, Amsterdam, 2000. doi:10.1016/B978-044482537-7/50004-8


If you have difficulty accessing that chapter, an earlier draft of the survey
is here: PDF download.