anchor.diff:  | Baseline | - | 193.8 us | -
   
   | http://ww1.microchip.com/downloads/en/DeviceDoc/KSZ8863MLL_FLL_RLL_DS.pdf[MICREL KSZ8863] (embedded in X5 actuator)
  -| 100 Mbit/s | +43.5 us | 3.4 us
  +| 100 Mbit/s | +43.5 us | 2.9 us
   
   | http://www.downloads.netgear.com/files/GDC/GS105/GS105_datasheet_04Sept03.pdf[NETGEAR ProSAFE GS105]
   | 1 Gbit/s | +51.0 us | 47 us
   
   | https://routerboard.com/RB750Gr2[MikroTik RB750Gr2 (RouterBOARD hEX)]
  -| 1 Gbit/s | +7.7 us | 3.7 us
  +| 1 Gbit/s | +7.7 us | 3.6 us
   
   |====
   
  
positive.code: 
  Whenever I get this far in a conversation, I usually get push back saying that this would of course be the case for single devices, but that UDP will blow up once more than one thing is connected.
  
  In order to test this hypothesis we put together a test setup consisting of 40 HEBI Robotics IO boards that are connected to a  http://www.downloads.netgear.com/files/GDC/GS748Tv1/GS748T_ds_03Feb05.pdf[GS748T] 48-port Ethernet switch.
  
  image::udp/multiple-boards.jpg[]
  
  The idea is to send a single broadcast message that gets received by all devices at the same time. This would then trigger 40x 266 byte (>10KB) return packets that often arrive at the switch within less than 250 nanoseconds. This would be representative of a very high degree of freedom system such as a full body humanoid robot.
  
  In order to keep the sender device from overloading, I changed the sender to a Desktop computer running https://ennerf.github.io/2016/09/20/A-Practical-Look-at-Latency-in-Robotics-The-Importance-of-Metrics-and-Operating-Systems.html[Scientific Linux 6.6 with real-time kernel]. Let's again take a look at the jitter of the underlying operating system. The figure below shows the difference between an idealized signal that ticks every 10ms and the actual measurements of the loop start times. It shows that this setup suffers from more than an order of magnitude more jitter than the embedded OS.
  
  [NOTE]
  The corresponding jHiccup control chart looks identical to the OS blog post.
  
  image::udp/os-jitter-linux-rt.png[]
  
  Gathering data at 100Hz close to 90 minutes resulted in more than 20 million returned packets. Again, there were no packet drops or re-ordering of packets from a single source. I also ran some tests at 1KHz and the result looks the same.
  
  image::udp/rtt-linux-40x-zoomed.png[]
  
  Rather than blowing up, what actually happens is that all packets get stored in the internal buffer of the switch (1.632MB), queued, and forwarded to the target port as fast as possible. Since the original sender is connected via 1 Gigabit, the packets arrive every ~2us. All of them have the same start timestamp, so the graph below shows this as vertical columns.
  
  image::udp/rtt-linux-40x.png[]
  
  What surprised me is that the latency is so much higher than the previous tests. Looking at the previous tests, I would have expected most columns to start at around ~180us and end at ~280us. While this is sometimes the case, the majority of columns start above 300 us. After some initial research I suspect that this is related to the Linux https://en.wikipedia.org/wiki/New_API[NAPI] using polling mode rather than working with interrupts.
  
  // Single unicast on LinuxRT has almost the same performance (minux 50us at the top). The 4 low bars may be polling intervals after the kernel driver switches to polling mode. Will take additional baseline-like data for Linux. From what I've read, sending should be non-blocking as long as the send buffer isn't full and the OS doesn't after the sys call.
  // See:
  
negative.code: 
  == Benchmark Setup
  
  A common way to benchmark networks is to setup two computers and have a sender transmit a message to a receiver that echoes it back. That way the sender can measure the round trip time and gather statistics of the network. This works well, but there is a lot of 'magic' that happens in the underlying operating system and it's device drivers.
  
  In order to mitigate this potential source of jitter, I setup a benchmark that measures the round trip time between two dedicated embedded systems. The overall approach remains the same, i.e.,
  
  1. Sender sends packet to receiver in fixed intervals
  2. Receiver echoes packet back to sender
  3. Sender measures the round trip time
  4. Measurement gets persisted to disk
  
  As the sender and receiver devices I'm using two HEBI Robotics I/O Boards. It's a product that we haven't advertised yet, but customers usually use it to integrate actuators with external devices or to get various sensor input into MATLAB. It has 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. It sports a 168MHz ARM microcontroller (STM32f407) and a 100 Mbit network port. In order to make sure that the devices are not doing anything else, I programmed custom firmware specifically for testing the network stack. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here]. The hardware timer (counter) resolution was set to 250ns.
  
  image::udp/io-boards.jpg[HEBI Robotics IO Board]
  
  Since storing multiple Gigabytes of data on an embedded device is challenging, I setup a https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[UDP server (Java)] that receives measurement data and persists the data to disk. The main socket handler writes incoming packets into a double buffered structure that can be persisted by a background thread without halting the packet handler. The synchronization between the threads is done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by a small amount of data.
  
  The resulting binary data was loaded into MATLAB(C) for analysis and to generate the plots. The code can be found https://gist.github.com/ennerf/19b48406a066f6e946a0567a1a4de1ed[here].
  
  == Ethernet Packet Structure
  
  UDP datagrams include various types of overhead in addition to the actual payload. The minimum overhead is 66 bytes. Additionally, Ethernet Frames have a minimum size of 84 bytes, which makes the minimum payload for a UDP datagram 18 bytes.
  
  .UDP/IP Packet Size
  [width="100%",options="header,footer",cols="a,a,a"]
  