What should this COM communication look like?
The low-level protocol seems to be fairly straightforward.

The host messages always start with 0x02, followed by two bytes of length (big-endian, most often 0x00 0x01 = 1). This is followed by length bytes of data (some sort of command byte or register address, sometimes with additional data) and a one-byte checksum of some sort.

The replies from the device have a similar structure. The first byte is a repeat of the command/address byte that the host sent, followed by two bytes of length, length bytes of data, and the checksum.

Beyond that, the meaning of the data fields is probably completely proprietary. Clearly it is sometimes ASCII strings, but a lot is binary data. You're going to have to spend a lot of time correlating the content of the messages with what you see in the user interface and whatever else might be happening to the device at the time.