1. A method comprising: generating, for each given sensor of a plurality of sensors for detecting objects in a vehicle's environment, a 3D model of the given sensor's field of view; aggregating, by one or more processors, the plurality of 3D models to generate a comprehensive model, wherein the comprehensive model indicates an extent of an aggregated field of view for the plurality of sensors; combining the comprehensive model with map information corresponding to environmental data for the vehicle's environment obtained at a previous point in time using probability data of the map information indicating a probability of detecting objects at various locations in the map information from various possible locations of the vehicle to produce a combined model annotated with information identifying a first portion of the environment as occupied by an object, a second portion of the environment as unoccupied by an object, and a third portion of the environment as unobserved by any of the plurality of sensors; and using the combined model to maneuver the vehicle.