1. A method for controlling an event structure system, the method comprising: recognizing multiple-person interaction primitives from an image, which is displayed on a display screen; composing an event by inference based on temporal relations using the multiple-person interaction primitive; and determining a final event by eliminating an unnecessary event from the composed event and adding a new event in the composed event after the unnecessary event is eliminated.