36. A non-transitory computer-readable medium comprising instructions for coding video data that, when executed, cause a processor to: maintain a plurality of context models for entropy coding transform coefficients of the video data, wherein the plurality of context models includes one or more context models that are each used for a different transform unit size and at least one joint context model used for two or more transform unit sizes; select the joint context model shared by a first transform unit having a first size with transform coefficients that are zeroed out to generate a retained coefficient block of a second size and a second transform unit having the second size for entropy coding transform coefficients in one of the first transform unit with the retained coefficient block or the second transform unit, wherein the first size and the second size are different, and wherein each of the transform units comprises a residual video data unit including a plurality of blocks of transform coefficients; select contexts for transform coefficients associated with the one of the first transform unit with the retained coefficient block or the second transform unit according to the joint context model, wherein the instructions cause the processor to assign a context in the joint context model to a given one of the transform coefficients based on values of previously coded neighboring transform coefficients of the one of the transform units, and determine a probability estimate for a value of the given one of the transform coefficients associated with the assigned context in the joint context model; and entropy code the transform coefficients associated with the one of the transform units using context adaptive binary arithmetic coding (CABAC) based on the selected contexts.
36. A non-transitory computer-readable medium comprising instructions for coding video data that, when executed, cause a processor to: maintain a plurality of context models for entropy coding transform coefficients of the video data, wherein the plurality of context models includes one or more context models that are each used for a different transform unit size and at least one joint context model used for two or more transform unit sizes; select the joint context model shared by a first transform unit having a first size with transform coefficients that are zeroed out to generate a retained coefficient block of a second size and a second transform unit having the second size for entropy coding transform coefficients in one of the first transform unit with the retained coefficient block or the second transform unit, wherein the first size and the second size are different, and wherein each of the transform units comprises a residual video data unit including a plurality of blocks of transform coefficients; select contexts for transform coefficients associated with the one of the first transform unit with the retained coefficient block or the second transform unit according to the joint context model, wherein the instructions cause the processor to assign a context in the joint context model to a given one of the transform coefficients based on values of previously coded neighboring transform coefficients of the one of the transform units, and determine a probability estimate for a value of the given one of the transform coefficients associated with the assigned context in the joint context model; and entropy code the transform coefficients associated with the one of the transform units using context adaptive binary arithmetic coding (CABAC) based on the selected contexts. 41. The non-transitory computer-readable medium of claim 36 , wherein the instructions comprise instructions for decoding video data, further comprising instructions that cause the processor to: receive a bitstream that represents encoded transform coefficients associated with the retained coefficient block within the first transform unit having the first size; select contexts for the encoded transform coefficients according to the joint context model; and entropy decode the encoded transform coefficients into the retained coefficient block within the first transform unit having the first size based on the selected contexts.