1. A method for a three-dimensional video encoding or decoding system, the method comprising: receiving input data associated with a motion vector or disparity vector of a current block; using a unified derivation method to derive a unified disparity vector from depth data of an associated depth block corresponding to the current block, wherein the unified derivation method generates the unified disparity vector from a subset of depth samples in the associated depth block, and the unified derivation method is applied in Inter mode, Skip mode, or Direct mode when one disparity vector derived from depth data is required for encoding or decoding, wherein the subset contains less depth samples than the associated depth block; and encoding or decoding the input data using said unified disparity vector in the Inter mode, Skip mode, or Direct mode.