   Using structs for network packets
  I was just curious if anybody else uses structures for their data packets? Previously i had an object that contained a byte array, and would just .add(data) to add something to the packet.Now that I've switched over to using structures I find it's much easier. Example...    '''<summary>This MSG is used to shutdown the server LOCALLY</summary>    Public Structure M00001   Public Command As Short'Command of this packet - Will be filled in automatically on send   Public PASS1 As Short   Public PASS2 As Short   Public PASS3 As Short    End Structure    '''<summary>This MSG is used to shutdown a NODE server REMOTELY</summary>    Public Structure M00002   Public Command As Short'Command of this packet - Will be filled in automatically on send   Public PASS1 As Short   Public PASS2 As Short   Public PASS3 As Short    End Structure    '''<summary>This MSG is used to register with the Node controller</summary>    Public Structure M01000   Public Command As Short'Command of this packet - Will be filled in automatically on send   Public PORT As Short   Public TYPE As Byte   <MarshalAs(UnmanagedType.ByValTStr, SizeConst:=32)> Public NAME As String   <MarshalAs(UnmanagedType.ByValTStr, SizeConst:=32)> Public IP As String    End StructureWhen the packet is sent the structure type for example "M01000" gets turned into a short and put into the "Command" property of the structure. Then I serialize the structure and send it. When it's received it reads the "command" property and creates a structure of the correct type. Then it deserializes the data into the structure.I find this works out alot better because I dont have to worry about how the data is added to the packet, I just have to read the structure property.Has anybody else done it this way or does anybody see any issues?   
  This method could cause problems when sending packets accross platforms, or even to the same program compiled by different compilers. This will be due to different OS/platforms/compilers packing data differently, using different endians etc..;  In my instance the server will all be in-house and rebuilds will all be pushed at the same time...However the client will be run on various windows machines, could it still be an issue there? Like for example a machine with XXX CPU will do this but YYY CPU will do that. Is there something I can test for, or would it be as simple as serializing something and then checking the result?   ;Quote:Original post by QuarantineHowever the client will be run on various windows machines, could it still be an issue there?As far as endianness goes, you'll be alright. As long as your clients and servers all run on x86/x64, that is. I don't know much about Visual Basic, but you should be able to rely on the ordering of fields in a structure. However, I cannot say the same thing for how the structure will be padded. If your data is organized inefficiently, or if it's not properly aligned (this may be out of your control) bad things might happen under different compiler versions or configurations. Dumping raw memory images into a network stream or file is generally discouraged. Serialize your data properly instead.EDIT: Also, since the size of a struct element is usually constant, you may find yourself wasting (or lacking), especially when it comes to sending strings.   ;  Strings are currently an issue because they have to be a fixed length. Currently I'm just sending the raw packet as is over the network. However there will eventually be a wrapper around it for reliable UDP as well as compression so it's negligible...Quote:Windryder:Dumping raw memory images into a network stream or file is generally discouraged. Serialize your data properly instead.I'm using the .NET marshaling functions to get a pointer to the structures memory and copy it directly into a byte array. The built-in .NET serializing functions add a lot of meta-data about the object being serialized, is there another way I can do this that I don't know about?    Public Function RawSerialize(ByVal anything As Object) As Byte()   Dim rawsize As Integer = Marshal.SizeOf(anything)   Dim buffer As IntPtr = Marshal.AllocHGlobal(rawsize)   Dim rawdatas(rawsize - 1) As Byte   Marshal.StructureToPtr(anything, buffer, False)   Marshal.Copy(buffer, rawdatas, 0, rawsize)   Marshal.FreeHGlobal(buffer)   Return rawdatas    End Function    Public Function RawDeserialize(ByVal rawdatas As Byte(), ByVal anytype As Type) As Object   Dim rawsize As Integer = Marshal.SizeOf(anytype)   Dim buffer As IntPtr = Marshal.AllocHGlobal(rawsize)   If (rawsize > rawdatas.Length) Then  Return Nothing   End If   Marshal.Copy(rawdatas, 0, buffer, rawsize)   Dim retobj As Object = Marshal.PtrToStructure(buffer, anytype)   Marshal.FreeHGlobal(buffer)   Return retobj    End Function;  In C#, you'd use the StructLayoutAttribute (to make it deterministic/sequential), and then use some unsafe code to pin the structure and copy to bytes.Something like this (written from memory):[StructLayout(StructLayoutKind.Sequential)]struct MyStruct {  public int fourBytes;  public short twoBytes;}byte[] GetBytes(ref StructLayoutKind data) {  unsafe {    fixed (StructLayoutKind *ptr = data) { byte[] ret = new byte[sizeof(data)]; for (int i = 0, n = ret.Length; i != n; ++i) {   ret = ((byte *)ptr); } return ret;    }  }}Hopefully you can do the same thing in VB. If not, you can write this part in C#, build a managed library, and add it as a reference in your VB project.Personally, I'd write a custom serializer based on System.Reflection, though.(I agree that the built-in serializer of .NET has way too much overhead)   ;  VB does have the sequential attribute and I've read it's on by default, but also that it's not, so I think I'm going to declare it on them just to be safe :)However I was looking at System.Reflection and I can visualize iterating through all the properties in the structure and packing them into a byte array. But I think that would be a step backwards, unless I'm missing something. Using reflection I would need to define where each property is in the packet.With the structure I don't need to know what data is where in the packet, I just have a struct on one end thats exactly the same on the other. I have a separate module that contains all the message structures that is used in both the client and server side, so they are always the same...Am I looking at using reflection in the wrong way?   ;Quote:Using reflection I would need to define where each property is in the packet.No, you just need to emit the marshaled data in a known order, and then read it back in in the same order on the other end. As long as the members in the struct are iterated in the same order (say, first to last), then you're good.I like using Reflection, because you can switch on the type. For things like string or byte[], a struct isn't good enough, because the actual data lives somewhere else on the heap, so just sending a struct won't be enough.;Quote:No, you just need to emit the marshaled data in a known order, and then read it back in in the same order on the other end. As long as the members in the struct are iterated in the same order (say, first to last), then you're good.But then wouldn't you have to do that for every packet type? So say you had 500 different types of packets, you would need a giant select statement based on the packet type? Then write code to read each individual type? Or am I not understanding you correctly?To me structures are all about having a fixed size and layout of data. From way back in the day using them as "records" in binary files in qbasic(wow feel old now). Originally back then you were REQUIRED to specify the length of a string, I know it's different now. If you try to marshal a structure with a NON-fixed length string in it all you get is garbage for the string. (I think it's a pointer to the actual string data/object) Also you can use the layoutkind.explicit and layoutkind.sequential to ensure they are exactly what is needed.Right now to convert the raw byte array back to the correct structure it's only a couple lines of code. For the names of the packet structures they are all in the format M00000. The five 0s being the command or "type" of the packet. For example a position update may be M06000...  PAYLOAD = UDPRecv.Receive(RP)   'Pull the next data packet  'BUILD the string to create the correct structure with  Dim INSTRUCTION As Short = BitConverter.ToInt16(PAYLOAD, 0)'GET the packet type  Dim STRNAME As String = "NodeControl.MessageStructures+M" + CStr(INSTRUCTION).PadLeft(5, "0")  Dim ty As Type = Type.GetType(STRNAME)  Dim PACKET = RawDeserialize(PAYLOAD, ty)  MSGQ.Enqueue(PACKET)And sending is...    Public Sub SendMSG(ByVal PACKET As Object, ByVal IP As String, ByVal PORT As Short)   PACKET.Command = CShort(Right(PACKET.GetType.ToString, 5))   Dim PAYLOAD() As Byte = RawSerialize(PACKET)   UDPSend.Send(PAYLOAD, PAYLOAD.Length, IP, PORT)    End SubThat gives me a perfect copy of the structure as it was sent... Again I know this only works if everything in the structure is fixed length, but thats how I've always used them anyways...Again though tell me if I'm not understanding your explanation of using reflection. In your quote above you mentioned using marshaling to read/write the data in a known order. Isn't that what I'm doing by reading/writing into a known fixed length structure?   ;Why wouldn't the natural order of the data members be enough? You don't need to specify that at all.Something like this:foreach (MemberInfo[] mi in data.GetType().GetMembers()){  visitor.Serialize(mi.MemberType, mi.GetValue(data));}Note that this can serialize any data type that contains only member types that you know how to serialize. No additional work needed! A simple, three-line function takes care of all of your compound data types.