1. A method implemented in a video coder for deriving a motion vector predictor (MVP) candidate set for a block, the method comprising: determining neighboring blocks of the block; generating a first MVP candidate set from MVP candidates associated with the neighboring blocks, wherein a fixed target size is determined for the first MVP candidate set; determining whether one or more redundant MVP candidates exist in the first MVP candidate set; generating a second MVP candidate set according to the first MVP candidate set, wherein said generating the second MVP candidate set comprises removing said one or more redundant MVP candidates from the first MVP candidate set if said one or more redundant MVP candidates exist; generating a third MVP candidate set if a size of the second MVP candidate set is smaller than the fixed target size, wherein said generating the third MVP candidate set comprises adding one or more pre-defined replacement MVP candidates to the second MVP candidate set until a size of the third MVP candidate set equal to the fixed target size; and encoding or decoding the block using the third MVP candidate set; wherein a value is assigned to each pre-defined replacement MVP candidate, wherein when the block is encoded or to be encoded by Merge mode, the value corresponds to one or a combination of prediction direction, reference index, and motion vector, and said one or more pre-defined replacement MVP candidates added to the second MVP candidate set comprise both a first zero vector with a reference frame index equal to 0 in list 0 and a second zero vector with a reference frame index equal to 0 in list 1, wherein when the block is encoded or to be encoded by Advanced Motion Vector Prediction (AMVP) mode, the value corresponds to motion vector (MV), and said one or more pre-defined replacement MVP candidates added to the second MVP candidate set comprise one or more zero vectors.