1. An apparatus for performing video segmentation, comprising: a plurality of image sensors of a first type; a plurality of second, auxiliary sensors of a second type; a processor executing computer-executable instructions stored in memory the instructions including: capturing red, green, and blue (RGB) image video data from the plurality of image sensors of the first type; capturing second video data from the plurality of second, auxiliary sensors of the second type; determining, for a reference frame, an initial segmentation map by segmenting a first object-of-interest from a background from the second video data; determining a history of segmentation information on a pixel-by-pixel basis, wherein a length of the history is based on motion data for the first object-of-interest; generating a refined segmentation map by refining on a pixel-by-pixel basis a corresponding value of the initial segmentation map based on the history of segmentation information; performing RGB segmentation on a corresponding reference frame of the RGB image video data based on the refined segmentation map; and performing segmentation of a second object-of-interest by: determining, for the reference frame, a second initial segmentation map by segmenting the second object-of-interest from a background from the second video data; generating a second refined segmentation map comprising the second object-of-interest by refining on a pixel-by-pixel basis a corresponding value of the initial segmentation map based on the history of segmentation information; and performing RGB segmentation on a corresponding reference frame of the RGB image video based on the second refined segmentation map.