1. A non-transitory, computer-readable storage medium storing program instructions that when executed by one or more computing devices implement: determining a segmentation shape prediction for an image of a video sequence based on received segmentation information; determining a segmentation color model for the image of the video sequence based on the received segmentation information for the image of the video sequence; automatically generating a segmentation of the image based on the segmentation shape prediction and the segmentation color model; and performing said automatically generating a segmentation for a plurality of subsequent images in the video sequence, wherein for each of the plurality of subsequent images the segmentation is based on the same segmentation color model and a different segmentation shape prediction updated for each of the plurality of subsequent images.