   Protocol design
  I'm currently developing an application that will require communications with a server over an (inter-) network connection back and forth. Currently, I have determined the amount of information that I need to send during different stages, such as identification, information requests(and the actual response) and pushed updates. However, I'm up to a point where I need to start looking into putting this together as a protocol that both my clients and the server understand. Sharing code between clients and server is not an option: I'm working with embedded clients and a normal server, so I am looking at different languages.All previous attempts at writing networked apps usually ended up in disaster because I skipped the more important part: Designing a proper protocol that's extensible in case it needs additions. Any tips for this? Perhaps reading material?Toolmaker   
  1) Take a long hard look at HTTP. If it can do the job (and it sounds as if it can), then use that. It's inherently extensible by adding query parameters and headers. It has lots of libraries for a variety of languages already written.2) If HTTP doesn't work, look for something pre-existing at a lower level. For example, Google Protocol Buffers is one of many IDLs that can generate binary serialization code for a variety of languages. It doesn't go all the way down to framing what packet is what, though -- while it can skip an unknown field, it can't tell the boundary between an unknown packet and the following known packet without extra framing. What I'd end up doing is wrapping the "message" protocol buffer in an "outer" protocol buffer that contains a type code (varint) and a "message" member that's the actual inner protocol buffer data.3) If you really don't want to use anything pre-made, then the base level of extensibility comes from being able to tell "what" something is on the wire, and being able to skip that "what" if you don't know how to handle it. The easiest way to do that is by sending packets as type-length-data. For example:PACKET := Typecode(2) Length(2) Data(N)"Typecode" is an unsigned short (2 bytes) telling you what the packet is."Length" is an unsigned short (2 bytes) telling you how much data follows (0 to 65535 bytes)."Data" is the actual "message data" of the packet.With this kind of definition, it's trivial for networking code to read two bytes to get the type, read two bytes to get the length, and then, based on the type code, figure out whether to skip N bytes, or read N bytes and dispatch to some code handler that knows how to handle packets of the given type.   ;  HTTP could be an idea. I'm not exactly writing a HTTP server, nor am I serving webpages, but I need to send some basic text data, such as identification codes and then regular updates with some doubles and ints(which I could easily parse server-side) and then return information based on that information. Sometimes I need to push basic information down to the client application, which mostly consists of text and sometimes some binary data. I've thought about using XML for it. Yes, it's extra overhead in context of the amount of data sent, but it's also very simple to use and pull apart on both the client and the server. The extra added overhead wouldn't be gigantic, and the amount of traffic generated by the system wouldn't be gigantic.Thoughts?   ;Quote:Original post by ToolmakerI've thought about using XML for it. Yes, it's extra overhead in context of the amount of data sent, but it's also very simple to use and pull apart on both the client and the server. The extra added overhead wouldn't be gigantic, and the amount of traffic generated by the system wouldn't be gigantic.Thoughts?I think this is a good idea. If what you need to do isn't time critical, parsing an xml is a trivial task. If you start now you can have it working by tonight.   ;Quote:Original post by owlQuote:Original post by ToolmakerI've thought about using XML for it. Yes, it's extra overhead in context of the amount of data sent, but it's also very simple to use and pull apart on both the client and the server. The extra added overhead wouldn't be gigantic, and the amount of traffic generated by the system wouldn't be gigantic.Thoughts?I think this is a good idea. If what you need to do isn't time critical, parsing an xml is a trivial task. If you start now you can have it working by tonight.You may want to check and see if you can actually get an XML parsing library running on your embedded clients before committing to XML. Personally, I'd rather use a plaintext HTTP transfer with key=value content than write an XML parser myself.   