1. A method for mapping a scene depicted in a sequence of video frames, the method comprising: segmenting a background image representing static elements of the scene into one or more spatially ordered regions, wherein each spatially ordered region has a corresponding a set of pixels in the background image; receiving a current video frame, of the sequence; identifying one or more foreground objects depicted in the current video, wherein each foreground object occludes the pixels in one or more of the spatially ordered regions of the background image; analyzing the one or more spatially ordered regions of the background image to identify at least a spatial ordering for one or more of the foreground objects, relative to the spatial ordering of the one or more spatially ordered regions of the background image; and generating, based on the spatially ordered regions of the scene and the spatial ordering for the one or more foreground objects, an annotated map of the scene specifying scene geometry for one or more of the spatially ordered regions of the scene.