1. A method of decoding a coded video bitstream with temporal motion vector prediction, the method comprising: parsing a flag from a header a sub-picture unit or a header of a picture of the coded video; and determining whether the flag indicates that temporal motion vector prediction is used or not used; wherein if the flag indicates that temporal motion vector prediction is used, the method further comprises: creating a first list of motion vector predictors comprising a plurality of motion vector predictors derived from at least one motion vector from a collocated reference picture; and parsing a first parameter from the coded video bitstream which indicates a selected motion vector predictor out of the first list for a prediction unit in the sub-picture unit; and wherein if a temporal layer of the picture is determined to be the lowest or base layer, the flag is set to indicate that temporal motion vector prediction is not used; otherwise, the flag is set to indicate that temporal motion vector prediction is used.