Is there a way to measure the amount of bytes that are possible to encode in a DNA molecule?
Unfortunately the answer is highly dependent on what you mean. In the simplest terms, comparing it directly to how we measure data storage in digital media, the number of different states of a DNA string of length $n$ can have is simply $4^n$. A byte holds $2^8$ different states so the number of bytes in a DNA string of length $n$ is $\frac{n}{4}$. Of course, actually accessing this information would be more difficult than simply having it in a strand of DNA.

DNA in real organisms is not random, however, so sequences are not randomly distributed meaning that you could compress that information down into less bits than that, Shannon information style.

However, you can decently argue that this doesn't really tell us how much information is in real DNA because real DNA has structure that matters. It has exons and introns and promoter regions and so on. Meanwhile, the sequences in protein coding regions are a lot more important than sequences in non-coding regions for the most part. Large parts of the genome are functionally irrelevant but because of this they tend to be more random and thus have higher Shannon information.