1. A method of processing video data, the method comprising: obtaining, from a bitstream that includes an encoded representation of the video data, a non-nested Supplemental Enhancement Information (SEI) message that is not nested within another SEI message in the bitstream; determining a layer of the bitstream to which the non-nested SEI message is applicable, wherein the non-nested SEI message is applicable to layers for which video coding layer (VCL) network abstraction layer (NAL) units of the bitstream have layer identifiers equal to a layer identifier of a SEI NAL unit that encapsulates the non-nested SEI message, and wherein a temporal identifier of the SEI NAL unit is equal to a temporal identifier of an access unit containing the SEI NAL unit; and processing, based in part on one or more syntax elements in the non-nested SEI message, video data of the layer of the bitstream to which the non-nested SEI message is applicable.