1. A method of decoding, in a video decoder, a block of enhancement-layer transform domain data for an encoded video and corresponding to a reconstructed block of reference-layer transform domain data, the method comprising: for an enhancement-layer element of the block of enhancement-layer transform domain data, wherein the block of enhancement-layer transform domain data is of size W×H, wherein the reconstructed block of reference-layer transform domain data is of size W/R×H/R, and wherein R comprises a non-unity scale factor, determining whether the enhancement-layer element is located in an upper-left W/R×H/R portion of the block of enhancement-layer transform domain data and, if the enhancement-layer element is within the upper left W/R×H/R portion, determining a context for that enhancement-layer element based, at least in part, upon a plurality of elements in the reconstructed block of reference-layer transform domain data without upscaling, wherein the plurality of elements are identified by a template that selects the plurality of elements in the reference-layer and wherein the position of the template in the reconstructed block of reference-layer transform domain data is based upon the position of that enhancement-layer element in the block of enhancement-layer transform domain data, and if the enhancement-layer element is not within the upper left W/R×H/R portion, determining the context for that enhancement-layer element using single-layer contexts; entropy decoding that enhancement-layer element using its determined context; and reconstructing and outputting for display a block of image data based on the block of enhancement-layer transform domain data.