1. An apparatus for video object segmentation, comprising: a memory and frame alignment mechanism that stores a plurality of frames of video and aligns one or more reference frames with a current frame containing a video object; circuitry that generates an intermediate mask for the current frame based on a neighborhood similarity metric; and a processor that refines the intermediate mask by using information from at least one video frame to generate a mask representative of the video object in the current frame: circuitry that generates initial foreground masks for a current frame with respect to each aligned reference frame based on a neighborhood similarity metric; and a generator that combines information from the initial foreground masks to generate an intermediate mask for the current frame before refining the intermediate foreground mask.