1. A method of annotating video images associated with an environmental situation based on detected actions of a human interacting with the environmental situation, the method comprising: receiving, with an information processing system, a set of real-time video images captured by at least one video camera associated with an environment presenting one or more environmental situations to a human; monitoring, with the information processing system, one or more user actions made by the human that is associated with the set of real-time video images with respect to the environmental situation; determining, based on the monitoring, that the human driver has one of performed and failed to perform at least one action associated with one or more images of the set of real-time video images; and annotating, with the information processing system, the one or more images of the set of real-time video images with a set of annotations based on the at least one action that has been one of performed and failed to be performed by the human.