what is the starting point of a contour as given by cvConvexityDefect() function in OpenCV
If you can obtain coordinates for the points you show in the right hand image, and you know the points relating to the shoulders, could you presume that the palms were further from the shoulder than the elbow is?

If you assume this to be true, then you simply need to find the distance from the shoulder to the elbow &amp; palm and call the point with the larger distance the palm.

Of course, you could bend your arm in such a way that the palm would be closer to the elbow, so perhaps this isn't an ideal method...