1. A computer-implemented method comprising: receiving, by one or more computing devices, sensor data identifying a first object and a second object in a vehicle's environment, the first and second objects being different; generating, by the one or more computing devices, a set of possible future trajectories of the first object based using map information describing the vehicle's environment and the sensor data; receiving, by the one or more computing devices, contextual information for the second object; determining, by the one or more computing devices, a likelihood value of each trajectory of the set of possible future trajectories based on the contextual information; determining, by the one or more computing devices, a final future trajectory based on the determined likelihood value for each trajectory of the set of possible future trajectories; and maneuvering, by the one or more computing devices, the vehicle in order to avoid the final future trajectory and the first object.