1. A computer-implemented method for tracking a physical target comprising: receiving a depth image that captures the physical target in a physical space; determining a first point based at least in part on an offset from a known point that corresponds to the physical target; determining a second point that corresponds to an average location of a limb of the physical target; determining a direction to search based on a line formed between the first point and the second point; and determining a third point of the limb based on the direction to search and depth values of the depth image.