anchor.diff:  
   == Conclusion
   
  -It seems that even consumer-grade Ethernet networks actually exhibit very deterministic performance with regards to latency. There were also no lost packets and none were received out-of-order. Large and dangerous industrial robots should still only be controlled using specialized networking equipment, but for most robotic applications I feel that standard UDP is more than sufficient.
  +It seems that even consumer-grade Ethernet networks actually exhibit very deterministic performance with regards to latency. There were also no lost packets and none were received out-of-order. Large and dangerous industrial robots should still only be controlled using specialized networking equipment, but for most robotic applications we feel that standard UDP is more than sufficient.
   
   The high latency and jitter that is commonly observed in network benchmarks seem to be caused almost entirely by the underlying operating systems and their device drivers. The two charts below show comparable baseline setups with different senders over time spans of 10 hours and 10 minutes. The configurations were as follows:
   
  
positive.code: // https://www.lantronix.com/resources/networking-tutorials/network-switching-tutorial/
  // http://www.embedded.com/design/connectivity/4023291/Real-Time-Ethernet
  
  Another problem that many roboticists are often concerned about is  https://en.wikipedia.org/wiki/Out-of-order_delivery[Out-of-Order Delivery], which says that a sequence of packets coming from a single source may be received in a different order. This is relevant for communicating over the internet, but generally does not apply to local networks without redundant routes and load balancing. Depending on the driver implementation it can theoretically happen on a local network, but I have yet to observe such a case.
  
  There are several competing networking standards that are built on Ethernet and can guarantee enough determinism to be used in industrial automation (https://en.wikipedia.org/wiki/Industrial_Ethernet[Industrial Ethernet]). They achieve this by enforcing tight control over the network layout and by limiting the components that can be connected. However, even cheap consumer grade network equipment can produce very good results if the network is controlled in a similar manner.
  
  == Benchmark Setup
  
  A common way to benchmark networks is to setup two computers and have a sender transmit a message to a receiver that echoes it back. That way the sender can measure the round trip time and gather statistics of the network. This generally works well, but large operating system stacks and device drivers can potentially add a lot of variation. In an attempt to reduce unwanted jitter, I decided to setup a benchmark using two embedded devices instead.
  
  As the sender and receiver devices I chose two HEBI Robotics I/O Boards.  We use it in conjunction with our actuators to interface with external devices, such as a gripper or pneumatic valve, or to get various sensor input into MATLAB. It has 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. It contains a 168MHz ARM microcontroller (STM32f407) and a 100 Mbit network port.
  
  [.text-center]
  .HEBI Robotics IO Board
  image::udp/io-board.jpg[link="../images/udp/io-board.jpg"]
  
  In order to isolate the network stack I wrote and programmed custom firmware based on http://www.chibios.org/[ChibiOS 2.6.8] and http://savannah.nongnu.org/projects/lwip/[lwIP 1.4.1]. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here]. The elapsed time was measured using a hardware counter with a resolution of 250ns. Since there was no way to store multiple Gigabytes of data on the devices, I needed a different solution for data logging.
  
  I decided store data remotely using a UDP service (see  https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[code]) that can receive measurement data and persist to disk. To avoid stalling the main socket handler in disk I/O, I had it write into a double buffered structure that got persisted by a background thread. The synchronization between the threads was done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by small amounts of data. It allows readers to flip buffers while keeping writers wait-free.
  
  // Since storing multiple Gigabytes of data on an embedded device is challenging, I also setup a https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[(Java) UDP server] that can receive measurement data and persist to disk. The main socket handler writes incoming packets into a double buffered structure that can be persisted by a background thread without halting the packet handler. The synchronization between the threads is done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by a small amount of data.
  
  // Alternatively: oscilloscope and logic analyzer
  
  
negative.code: In order to test the actual behavior we put together a test setup consisting of 40 HEBI Robotics I/O boards that are connected to a  http://www.downloads.netgear.com/files/GDC/GS748Tv1/GS748T_ds_03Feb05.pdf[GS748T] 48-port Ethernet switch. Each board runs the 'receiver' firmware. Sending a single broadcast message to all receiver devices triggers 40x266 byte (>10KB total) return packets that can arrive at the switch within less than 250 nanoseconds. This https://en.wikipedia.org/wiki/Micro-bursting_(networking)[Microburst] traffic pattern is representative of a high degree of freedom system such as a full body humanoid robot without doing any daisy-chaining.
  
  [.text-center]
  .Network test setup with 40 HEBI Robotics I/O Boards
  image::udp/multiple-boards.jpg[link="../images/udp/multiple-boards.jpg"]
  
  [NOTE]
  