What is "M" in Hartley's law?
\$M\$ is the number of distinct messages that can be transmitted per symbol. The number of bits per symbol is then \$\log_2 M\$.

For example, if you take a simple protocol that is either high (5V) or low (0V), then \$M=2\$ and each symbol transmits one bit.

Or, perhaps instead you can signal one of four things:


0V (00)
1.67V (01)
3.33V (10)
5V (11)


Now \$M=4\$, and each time you do this you transmit 2 bits of information.

If you take an extreme example, say you have this signal going into a 32-bit analog-to-digital converter. Now there are in theory \$M = 2^{32} = 4294967296\$ possible messages, and each one transmits 32 bits of information.

Of course at some point your ability to resolve the difference between each message is limited by noise, and this is what the Shannon-Hartley theorem is about.