Affine Transformation for matching 2D point sets
There are two ways by which you can approach this problem:


Match the point clouds directly with RANSAC.
Consider matching the resulting graphs (closely related to 1, still)


What you have done already is closest to #1. To complete it, you need to repeatedly obtain subsamples, find the transformation and then check its "error". The "error" is how many projected points are within a distance threshold (which you are going to have to estimate). For more information, please see this thesis. It does a great job at defining the problem in the beginning (which is very important) and then talking about the approaches.

But, since you are already dealing with pose estimation, you can incorporate its constraints into the model and go for estimation via graph matching. In this case, you would be looking at extracting a graph from your images and then compare it with the pose graph, OR, compare the extent to which two graphs coincide (the graph obtained from the left image to the graph obtained from the right image). A key part of this is the derivation of image skeletons. Gonzalez' book has a very good section on skeletons (chapter 11) and their applications.

The existence of a graph imposes certain constraints on the relationships between the points. For example, the hand "leafs" are expected to be no more than 5-7 "hops" from the root node (e.g. the head node) and this can improve the detection rate. For more information please see this paper. Or more generally look for "graph based pose estimation" and "graph recognition". Be aware that "graph matching" as a term is associated with something entirely different in mathematics.

Hope this helps.