1. A method of coding video data comprising: maintaining a plurality of context models for entropy coding transform coefficients of the video data, wherein the plurality of context models includes one or more context models that are each used for a different transform unit size and at least one joint context model used for two or more transform unit sizes; selecting the joint context model shared by a first transform unit having a first size with transform coefficients that are zeroed out to generate a retained coefficient block of a second size and a second transform unit having a second size for entropy coding transform coefficients in one of the first transform unit with the retained coefficient block or the second transform unit, wherein the first size and the second size are different, and wherein each of the transform units comprises a residual video data unit including a plurality of blocks of transform coefficients; selecting contexts for the transform coefficients associated with the one of the first transform unit with the retained coefficient block or the second transform unit according to the joint context model, wherein selecting contexts for the transform coefficients comprises assigning a context in the joint context model to a given one of the transform coefficients based on values of previously coded neighboring transform coefficients of the one of the transform units, and determining a probability estimate for a value of the given one of the transform coefficients associated with the assigned context in the joint context model; and entropy coding the transform coefficients associated with the one of the transform units using context adaptive binary arithmetic coding (CABAC) based on the selected contexts.