1. A method implemented by one or more processors, comprising: accessing a spatio-temporal model stored in one or more computer readable media, the spatio-temporal model generated based on received observations from multiple robots operating in an environment over a period of time, and the spatio-temporal model defining, for each of a plurality of objects in the environment, a plurality of pose values and corresponding times for the pose values; identifying an object of the objects in the environment, and a target time; selecting, from the spatio-temporal model, a group of pose values for the object based on proximity of the target time to the corresponding times for the pose values of the group; and generating a pose for the object for the target time based on the selected group of the pose values.