1. An image processing apparatus, comprising: a receiver configured to receive an image comprising color information of pixels and an initial disparity map comprising disparity information of the pixels, wherein disparity information for at least some of the pixels in the initial disparity map is undetermined; a constraint determining processor configured to determine a constraint; and a generator configured to generate a final disparity map by assigning similar disparities to two pixels that are adjacent to each other and that have similar colors, among the pixels, under the determined constraint, the final disparity map comprising a disparity value for each of the pixels.