1. A system comprising one or more computing devices, having one or more processors, the one or more computing devices being configured to: store behavior model data for a plurality of object classifications; receive object data from a monitoring source, wherein the object data comprises world-view data and actions data for a detected object; determine the detected object's classification based on the object data; access a first behavior model from the stored behavior model data, wherein the first behavior model is accessed based on the detected object's classification; filter the actions data for the detected object to identify actions of interest; update the first behavior model based on the world-view data and the identified actions of interest for the detected object, so as to generate an updated behavior model, wherein the updated behavior model is used for predicting how other objects, having a same classification as the detected object, will behave; and use the first behavior model to autonomously control a vehicle.