1. A method for mapping a scene depicted in a sequence of video frames, the method comprising: generating, from the sequence of video frames, a background image representing static elements in the scene; receiving a current video frame; identifying a plurality of regions of the current frame depicting the background image; analyzing the plurality of regions of the current frame to identify at least a first region of the background image of the scene that occludes at least a portion of a second region of the background image of the scene; and updating an annotated map specifying geometry for one or more of the plurality of regions in the background image to indicate that the first region is in front of the second region, relative to a position of a camera capturing the sequence of video frames.