1. A system for recognizing and representing activities in a video sequence, the system comprising: a computer processor adapted to execute computer modules, the computer modules comprising: an action detector, adapted to detect activities in frames of the video sequence, and generate an observation probability for each detected action; an activity information network, adapted to encode prior information about an activity domain, the activity domain being selected based on the detected actions of the video sequence and the prior information including contextual cues enforcing temporal constraints of each detected action and ordering constraints between the detected actions; and an activity inference engine, adapted to estimate, for each frame, based on the detected actions at a time instance in the frame and the prior information of the selected activity domain, a likely activity state, the likely activity state being a current activity and a plurality of activities likely having occurred prior to the current activity.