1. An image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera utilized for calibrating a panoramic image shot by a panoramic fish-eye camera to a 3D panoramic image, wherein the panoramic fish-eye camera comprises four fish-eye lens, four CMOS sensor modules and a depth calibration module, comprising the following steps: establishing a panoramic optical target space, wherein the panoramic optical target space comprises a depth calibration target; utilizing the panoramic fish-eye camera for shooting the panoramic image of the panoramic optical target space; establishing a depth calibration parameter between the depth calibration module and the depth calibration target, wherein the depth calibration parameter is a reference datum for calibrating depth; establishing an internal calibration parameter of the panoramic fish-eye camera, wherein the internal calibration parameter is the coordinate transformation parameter between the fish-eye lens and the CMOS sensor modules of the panoramic fish-eye camera; establishing an image stitching parameter of the panoramic image and the panoramic optical target space, wherein the image stitching parameter is used for a panoramic image stitching parameter by means of computing the relationships between the physical body and the space coordinate of the four fish-eye lens from the images shot by the panoramic fish-eye camera; establishing a space depth transformation parameter of the panoramic image and the panoramic optical target space, wherein the space depth transformation parameter is a transformation parameter between a 2D planar image and an object depth in 3D space; and utilizing the image stitching parameter, the space depth transformation parameter, the internal calibration parameter and the depth calibration parameter to calibrate the panoramic image for generating a 3D panoramic image.