1. A method of performing joint depth estimation and semantic labeling of an image by one or more computing devices, the method comprising: estimating global semantic and depth layouts of a scene of the image through machine learning by the one or more computing devices; estimating local semantic and depth layouts for respective ones of a plurality of segments of the scene of the image through machine learning by the one or more computing devices; and merging the estimated global semantic and depth layouts with the estimated local semantic and depth layouts by the one or more computing devices to semantically label and assign a depth value to individual pixels in the image.