anchor.diff:  
   == Switching Cost
   
  -As mentioned in the intro, most modern switches use the 'store-and-forward' approach that requires the switch to fully receive a packet before forwarding it appropriately. Therefore, the latency cost per switch is the time it takes a packet on the wire plus any switching overhead. The wire time is constant (2.03us or 20.3us for 266 bytes), but the overhead depends on the switch implementation. There is not much 3rd party data out there, so depending on your requirements you may need to conduct your own benchmarks if you need to evaluate hardware.
  +As mentioned in the background section, most modern Switches use the 'store-and-forward' approach that requires the Switch to fully receive a packet before forwarding it appropriately. Therefore, the latency cost per switch is the time it takes a packet on the wire plus any switching overhead. The wire time is constant (2.03us or 20.3us for 266 bytes), but the overhead depends on the switch implementation. It can be difficult to find good performance data for specific devices, so depending on your requirements you may need to conduct your own benchmarks if you need to evaluate hardware.
   
   [.text-center]
   .Benchmark setup using additional Switch
   image::udp/setup-switching-cost.png[link="../images/udp/setup-switching-cost.png"]
   
  -For this benchmark I tested three switches that were individually added to the baseline setup. In total there were about 1 million packets.  Over the course of all of our tests no packets were dropped or received out of order.
  +For this benchmark we tested the three following Switches and added them individually to the baseline setup as shown above,
  +
  +* http://ww1.microchip.com/downloads/en/DeviceDoc/KSZ8863MLL_FLL_RLL_DS.pdf[MICREL KSZ8863] (embedded in X5 actuators)
  +* http://www.downloads.netgear.com/files/GDC/GS105/GS105_datasheet_04Sept03.pdf[NETGEAR ProSAFE GS105]
  +* https://routerboard.com/RB750Gr2[MikroTik RB750Gr2 (RouterBOARD hEX)] (technically a Router, but disabling DHCP makes it act similar to a Switch)
  +
  +
  +In total there were about 1 million packets. Again, we did not observe any packet loss or out-of-order delivery.
   
   [.text-center]
   .Comparison of round trip time through different switches (35min)
   image::udp/rtt-switch-comparison.png[link="../images/udp/rtt-switch-comparison.png"]
   
  -Mode 2 (at 210 us) seems to disappear for higher round trip times, indicating an issue at the receiving step of the sender. Modes 1 and 3 do not seem to be affected by additional switches, indicating that they are caused by something happening on on the firmware side. The figure below shows a zoomed view of the time series highlighting the added jitter characteristics.
  +The figure below shows a zoomed view of the time series highlighting the added jitter characteristics. Modes 1 and 3 do not seem to be affected by additional switches. Mode 2 remains constant at 210 us and disappears for higher round trip times, indicating an issue at the receiving step of the sender.
   
   [.text-center]
   .Zoomed in view of switch comparison (10min)
   image::udp/comparison-switch-latency.png[link="../images/udp/comparison-switch-latency.png"]
   
  -Both KSZ8863 and the RB750Gr2 add a constant switching latency of 2.9 us and 3.6 us in addition to the wire time of 40.6 us and 4.06 us respectively to the RTT. The added jitter seems to be negligible at well below 1us.
  +Both KSZ8863 and the RB750Gr2 add a constant switching latency of 2.9 us and 3.6 us in addition to the wire time of 40.6 us and 4.06 us respectively to the RTT. The added jitter seems to be negligible at well below 1us. Surprisingly, the GS105 seems to have problems with this use case, resulting in higher latency and more jitter than the KSZ8863 even though it was connected using Gigabit. More details are in the table below.
   
   [width="100%",options="header",cols="3a,1a,1a,1a"]
   |====
  
positive.code: // https://www.lantronix.com/resources/networking-tutorials/network-switching-tutorial/
  // http://www.embedded.com/design/connectivity/4023291/Real-Time-Ethernet
  
  Another problem that many roboticists are often concerned about is  https://en.wikipedia.org/wiki/Out-of-order_delivery[Out-of-Order Delivery], which says that a sequence of packets coming from a single source may be received in a different order. This is relevant for communicating over the internet, but generally does not apply to local networks without redundant routes and load balancing. Depending on the driver implementation it can theoretically happen on a local network, but I have yet to observe such a case.
  
  There are several competing networking standards that are built on Ethernet and can guarantee enough determinism to be used in industrial automation (https://en.wikipedia.org/wiki/Industrial_Ethernet[Industrial Ethernet]). They achieve this by enforcing tight control over the network layout and by limiting the components that can be connected. However, even cheap consumer grade network equipment can produce very good results if the network is controlled in a similar manner.
  
  == Benchmark Setup
  
  A common way to benchmark networks is to setup two computers and have a sender transmit a message to a receiver that echoes it back. That way the sender can measure the round trip time and gather statistics of the network. This generally works well, but large operating system stacks and device drivers can potentially add a lot of variation. In an attempt to reduce unwanted jitter, I decided to setup a benchmark using two embedded devices instead.
  
  As the sender and receiver devices I chose two HEBI Robotics I/O Boards.  We use it in conjunction with our actuators to interface with external devices, such as a gripper or pneumatic valve, or to get various sensor input into MATLAB. It has 48 pins that serve a variety of functions (analog/digitial I/O, PWM, Encoder input, etc.) that can be accessed remotely via network. It contains a 168MHz ARM microcontroller (STM32f407) and a 100 Mbit network port.
  
  [.text-center]
  .HEBI Robotics IO Board
  image::udp/io-board.jpg[link="../images/udp/io-board.jpg"]
  
  In order to isolate the network stack I wrote and programmed custom firmware based on http://www.chibios.org/[ChibiOS 2.6.8] and http://savannah.nongnu.org/projects/lwip/[lwIP 1.4.1]. The relevant code pieces can be found https://gist.github.com/ennerf/36a57d432bcff20a58efcdee10f91bd9[here]. The elapsed time was measured using a hardware counter with a resolution of 250ns. Since there was no way to store multiple Gigabytes of data on the devices, I needed a different solution for data logging.
  
  I decided store data remotely using a UDP service (see  https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[code]) that can receive measurement data and persist to disk. To avoid stalling the main socket handler in disk I/O, I had it write into a double buffered structure that got persisted by a background thread. The synchronization between the threads was done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by small amounts of data. It allows readers to flip buffers while keeping writers wait-free.
  
  // Since storing multiple Gigabytes of data on an embedded device is challenging, I also setup a https://gist.github.com/ennerf/0ddc4396d15852d28e4eca4a8a923eb7[(Java) UDP server] that can receive measurement data and persist to disk. The main socket handler writes incoming packets into a double buffered structure that can be persisted by a background thread without halting the packet handler. The synchronization between the threads is done using a http://stuff-gil-says.blogspot.com/2014/11/writerreaderphaser-story-about-new.html[WriterReaderPhaser], which is a synchronization primitive that is very useful for persisting events that are represented by a small amount of data.
  
  // Alternatively: oscilloscope and logic analyzer
  
  
negative.code: // GS105: 244.8 us
  // RB750Gr2: 201.5 us
  
  According to the http://www.downloads.netgear.com/files/GDC/GS105/GS105_datasheet_04Sept03.pdf[GS105 spec sheet], the added network latency should be below 10us (1 Gbit) and 20us (100 Mbit). We did additional tests and it did seem to perform according to spec when using exclusively 100 Mbit/s or 1 Gbit/s on all ports. 
  
  We also conducted another baseline test that replaced the GS105 with a RB750Gr2. While we found a consistent improvement of 0.5us, we did not consider this significant enough to rerun all tests.
  
  == Scaling to Many Devices
  
  Another major concern about UDP is how performs with multiple devices on the network, rather than single point-to-point connections.
  
  