1. A system for recognizing and representing activities in a video sequence, the system comprising: an action detector, adapted to detect activities in frames of the video sequence, and generate an observation probability for each detected action; an activity information network, adapted to encode prior information about an activity domain, the activity domain being selected based on the detected actions of the video sequence; and an activity inference engine, adapted to estimate, for each frame, based on the detected actions in the frame and the prior information of the selected activity domain, a likely activity state, the likely activity state being a current activity and a plurality of activities likely having occurred prior to the current activity.