1. A three-dimensional information processing apparatus for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising: image sensing means for sensing images of the object from a plurality of coordinate positions using an image sensing system having one or a plurality of optical systems; information extraction means for extracting a plurality of depth information from image sensing related information obtained by said image sensing means at the plurality of coordinate positions; and conversion/unification means for converting and unifying the plurality of depth information extracted by said depth information extraction means into depth information expressed by a unified coordinate system.