1. A gaze prediction system for processing a collection of input facial images of a user to generate a system output that predicts a gaze position of the user, the gaze prediction system comprising: an image collection system implemented by one or more computers and configured to: obtain one or more calibration images of the user and, for each of the calibration images, a respective calibration label that labels a known gaze position of the user in the calibration image, after obtaining the one or more calibration images, obtain a query image of the user, and generate a neural network input that includes (i) the query image of the user, (ii) the one or more calibration images of the user, and (iii) the respective calibration labels for the one or more calibration images, and a neural network, wherein the neural network is implemented by the one or more computers and comprises one or more neural network layers, and wherein the neural network is configured to, at run time and after the neural network has been trained: process the neural network input comprising (i) the query image of the user, (ii) the one or more calibration images of the user, and (iii) the respective calibration labels for the one or more calibration images using the one or more neural network layers to generate a neural network output that characterizes a gaze position of the user in the query image.