1. A method of decoding video data comprising: maintaining a joint context model shared by a transform unit having a first size with coefficients that are zeroed out to generate a retained coefficient block and a transform unit having the second size, wherein the first size and the second size are different; selecting contexts for coefficients associated with the transform unit having one of the first size with the retained coefficient block and the second size according to the joint context model; and entropy coding the coefficients associated with the transform unit according to a coding process based on the selected contexts.