1. An apparatus for performing video segmentation, comprising: a plurality of image sensors of a first type; a plurality of second, auxiliary sensors of a second type; a processor executing computer-executable instructions stored in memory, said instructions including: capturing RGB image video data from a plurality of image sensors of a first type; capturing second video data from a plurality of second, auxiliary sensors of a second type; for a reference frame, determining an initial segmentation map by segmenting a first object-of-interest from a background from said second video data; determining a history of segmentation information on a pixel-by-pixel basis, wherein a length of said history is based on motion data for said first object-of-interest; generating a refined segmentation map by refining on a pixel-by-pixel basis a corresponding value of said initial segmentation map based on said history of segmentation information; and performing RGB segmentation on a corresponding reference frame of said RGB image video based on said refined segmentation map.