1. A system for real-time cascaded object recognition, the system comprising: a processor; and a memory, including instructions, which when executed on the processor, cause the processor to perform operations comprising: accessing image data at the system, the image data of an environment around the system, the image data is captured by a camera system that includes a depth camera; determining a set of regions in the image data by using a depth segmentation process that segments the set of regions from other regions in the image data based on depth data obtained by the depth camera, the set of regions including candidate objects; transmitting a subset of the image data corresponding to the set of regions to a remote server, the remote server to analyze the subset of the image data and detect an object in the subset of the image data, wherein the subset of the image data comprises pixels less than a threshold distance away from the camera system as measured by the depth camera; and receiving at the system from the remote server, an indication of the object detected in the subset of the image data.