1. A computer-implemented method comprising: identifying, by one or more computing devices, an object in a vehicle's environment, the object having a heading and location; generating, by the one or more computing devices, a set of possible actions for the object using map information describing the vehicle's environment and the heading and location of the object; generating, by the one or more computing devices, a set of possible future trajectories of the object based on the set of possible actions; receiving, by the one or more computing devices, contextual information including a status of the detected object; determining, by the one or more computing devices, a likelihood value of each trajectory of the set of possible future trajectories based on the contextual information; determining, by the one or more computing devices, a final future trajectory based on the determined likelihood value for each trajectory of the set of possible future trajectories; and maneuvering, by the one or more computing devices, the vehicle in order to avoid the final future trajectory and the object.