What is the name of the domain or theoretical problem of transmitting game states in an online game
I'm not sure there is a single "name" for this non-trivial situation -- not every situation you will run into will have a single "name" or terminology for that situation -- but there are multiple topics and concepts that seem relevant:


As Draconis mentions, serialization or pickling are relevant for how to convert a complex data structure in memory into a sequence of bits that can be sent to another party.
Data synchronization and file synchronization may also be relevant; there are techniques that may be useful if the data structure representing game state changes slowly, as then it may be more efficient to send only the changes to the other parties without having to repeatedly send the current state.  You might enjoy reading about rsync and mosh.
Video games may use various methods for lag compensation that are worth knowing about.


Stack Exchange has an entire site for game development.  I suggest spending some quality time there perusing questions that seem related, to get a taste of some of the methods, concepts, and techniques that are relevant.