What does DAC resolution mean?
"Does it mean it can generate 1024 voltage levels at the sample rate which is 12 GSa/s?" : yes, although with limits on analog bandwidth. If you ask it to generate alternating 0 / 1023 levels, for a 6GHz signal, it will be attenuated a bit as the rated output bandwidth is only 5GHz.

"Can it really do communications at 120 Gb/s or what would be the practical limitation?" : yes, via a number of parallel digital links that operate at a more reasonable 375Mbit each.