1. A method for use with depth images, comprising: (a) obtaining a depth image and information that specifies that a subset of pixels within the depth image correspond to a user; (b) identifying one or more spans of pixels, within the subset of pixels specified as corresponding to the user, that are potentially part of a hole in the subset of pixels that resulted from a portion of the user occluding another portion of the user when a capture device was used to obtain the depth image or a higher resolution version of the depth image; (c) analyzing span adjacent pixels to identify, as also potentially being part of a hole in the subset of pixels specified as corresponding to the user, each span adjacent pixel that both includes (c1) at least a first threshold number of neighboring pixels identified as potentially being part of a hole, and (c2) no more than a second threshold number of neighboring pixels identified as a potential hole boundary; and (d) grouping together pixels that are adjacent to one another and have been identified as potentially being part of a hole in the subset of pixels specified as corresponding to the user, to thereby identify one or more islands of pixels that potentially correspond to one or more holes in the subset of pixels specified as corresponding to the user.