1. A computer-implemented method of image segmentation comprising: receiving, at a processor, an image of a scene comprising a plurality of image elements, each image element having an associated color value and an associated depth value representing a distance between from an image sensor to a scene element: using the depth values to derive a set of three-dimensional planes present within the scene; and for each image element, determining whether the image element belongs to a plane from the set and labeling the image element accordingly, wherein the determining comprises evaluating a cost function over the set of planes having terms dependent on the depth value of the image element, and the color value of the image element and at least one neighboring image element.