1. A method comprising: decoding, by a processor in response to instructions stored on a non-transitory computer readable medium, a current block from a current frame from an encoded video stream, wherein the current block has a current block size, and wherein decoding the current block includes: determining a probability for entropy decoding a transform partitioning flag for the current block from the encoded video stream, based on a transform block size of an adjacent block that is available for decoding the current block, entropy decoding the transform partitioning flag for the current block using the probability, generating a reconstructed block based on the current block, wherein generating the reconstructed block includes: on a condition that the transform partitioning flag indicates that transform partitioning is omitted for the current block, generating a residual block by inverse transforming transform coefficients decoded from the encoded video stream for the current block, wherein the inverse transforming includes using a current block size inverse transform; on a condition that the transform partitioning flag indicates transform partitioning for the current block: identifying sub-blocks from the current block, wherein each sub-block from the sub-blocks has a respective sub-block size that is smaller than the current block size, and generating the residual by inverse transforming transform coefficients decoded from the encoded video stream for the respective sub-blocks, wherein the inverse transforming includes using sub-block size inverse transforms; and generating the reconstructed block based on the residual block, and including the reconstructed block in an output video stream; and outputting the output video stream.