1. An eye tracking system comprising: a video capture component configured to provide a video of an environment and a set of gaze tracking coordinates representing, for each of a plurality of frames comprising the video, a position within the environment at which a user is looking; a library, configured to store a plurality of feature sets representing respective areas of interest, with each area of interest including an object label representing an object of interest within the environment; an image annotation component configured to extract a plurality of features from a region of interest, defined around the gaze coordinates for a given frame, and match the extracted features to an area of interest in the library to produce a selected area of interest and a confidence value; and a verification component configured to accept the selected area of interest if the confidence value meets a threshold value and send the region of interest to a human expert at a user interface to assign an object label to the region of interest if the confidence value does not meet the threshold value; wherein the library is dynamically updated in response to input from the human expert to add new areas of interest and new object labels to the library.