1. A computer implemented method of simplifying updating of a predictive model using clustering observed points, the method including: observing a set of points in a three-dimensional (3D) sensory space; determining surface normal directions from the points; clustering the points by their surface normal directions and adjacency; accessing a predictive model of a hand; refining positions of segments of the predictive model; matching the clusters of the points to the segments; and using the matched clusters to refine the positions of the matched segments.