1. A method of tracking a target, the method comprising: receiving from a source one or more observed depth images of a scene including the target; labeling each pixel of the one or more observed depth images as either a foreground pixel belonging to the target, a foreground pixel belonging to another target, or a background pixel not belonging to a target; labeling each foreground pixel with body part information indicating a likelihood that that foreground pixel belongs to one or more body parts of the target; and modeling the target with a virtual skeleton including a plurality of virtual skeletal points, each virtual skeletal point including a three dimensional position derived from body part information of one or more foreground pixels from the one or more observed depth images.