why are 1B/2B/7B DiTs as hard to scale as 80B LLMs?

the weights to activations ratio is the same, the “sequence length” is smaller than most LLM context lengths, so what exactly is consuming memory in DiTs and even UNets?