1. A gaze prediction system for processing a collection of input facial images of a user to generate a system output that predicts a gaze position of the user, the gaze prediction system comprising: a neural network comprising one or more neural network layers, wherein the neural network is configured to: obtain a collection of input facial images of the user, wherein the collection of input facial images of the user comprises (i) a query image of the user, (ii) one or more calibration images of the user, and (iii) a respective calibration label that labels a known gaze position of the user for each of the one or more calibration images of the user; and process the received collection of input facial images of the user using the one or more neural network layers to generate a neural network output that characterizes a gaze position of the user in the query image.