1. A method of tracking pose of a hand or body comprising: receiving a stream of images depicting the hand or body; inputting at least part of one of the images to a plurality of layers of a prediction engine to obtain a distribution over poses of the hand or body; accessing a pool of candidate poses of the hand or body; and periodically replacing a sub-set of the candidate poses in the pool with samples from the distribution over poses of the hand or body.