1. A computer-implemented method for maneuvering a vehicle, the method comprising: receiving, by one or more computing devices, sensor data identifying a first object in a vehicle's environment; generating, by the one or more computing devices, a set of possible future trajectories of the first object based on the sensor data for the first object; determining, by the one or more computing devices, contextual information for a second object in the vehicle's environment based on the sensor data, the second object being different from the first object; determining, by the one or more computing devices, a final future trajectory based on the contextual information and a likelihood value for each possible future trajectory set of possible future trajectories, each likelihood value corresponding to a likelihood that the first object will follow a given possible future trajectory; and maneuvering, by the one or more computing devices, the vehicle in order to avoid the final future trajectory and the first object.