1. A computer-implemented method of image segmentation comprising: receiving, at a processor, an image of a scene comprising a plurality of image elements, each image element having an associated color value and an associated depth value representing a distance between from an image sensor to a scene element; applying a weighted line scoring calculation to a plurality of straight lines extracted from the image to determine the principal directions for the image, the weighted line scoring calculation accounting for at least the surface normal of an image element, the direction of a straight line associated with the image element, and a plurality of predetermined weights; using the depth values to derive a set of three-dimensional planes present within the scene; and for each image element, determining whether the image element belongs to a plane from the set and labeling the image element accordingly, wherein the determining comprises evaluating a cost function over the set of planes having terms dependent on the depth value of the image element, the cost function having terms at least related to the number of planes, the set of derived planes, a unary term related to the 3D coordinates of the image element, and a unary term related to the normal of the image element, and the color value of the image element and at least one neighboring image element.