anchor.diff:  
   // NOTE: TODO: give OSX example for mean/std-dev and let readers guess the maximum?. Re-read https://www.azul.com/giltene-how-java-got-the-hiccups/ .
   
  -Latency is an important practical concern in the robotics world that is often poorly understood. I've spent many hours looking for information on the latency characteristics of various robotic components, but have found very little informative data out there. Most benchmarks focus on the maximum throughput and either completely neglect latency, or measure it incorrectly. 
  +This is the first in a series of blog posts where I will try to share some of my own impressions and findings that have stemmed from several years of creating tools for robotics research. 
   
  -This post will hit 2 main topics: 
  +//Today's post will focus on operating systems and measuring latency.
  +
  +Communication latency is an important practical concern in the robotics world that is often poorly understood. I feel that a better understanding of latency can help robotics researchers and engineers make design and architecture decisions that greatly streamline accelerate the R&D process. I've personally spent many hours looking for information on the latency characteristics of various robotic components, but have had difficulty finding anything that is clearly presented or backed by solid data. From what I've found, most benchmarks focus on the maximum throughput and either ignore the subject of latency or measure it incorrectly. 
  +
  +Because of this my first post is on the topic of latency and will cover two main topics: 
   
   1. The details of how you measure latency matters.
   
   2. The OS that you use affects your latency.
   
  -My own background is in academic research. I've spent several years as a staff software engineer at the Robotics Institute at Carnegie Mellon University and am a co-founder of HEBI Robotics, a startup developing modular robotic components. We've worked on many different types of robots, including collaborative arms, wheeled robots, walking robots and snake robots.
  +My own background is in academic research. I've spent several years as a staff software engineer at the Robotics Institute at Carnegie Mellon University and I am a co-founder of HEBI Robotics, a startup developing modular robotic components. The teams I've been part of have worked on many different types of robots, including https://youtu.be/heOXsEnGb20[collaborative manipulators], https://www.youtube.com/watch?v=zaPtxre4tFc[wheeled robots], https://www.youtube.com/watch?v=7Mh3kqxle1c[walking robots] and https://www.youtube.com/watch?v=DUgt3NwzN-c[snake robots].
   
  -In general, our experience so far has been that robotics research can be greatly accelerated by designing systems that relax the hard real-time requirements on the software that is exposed to users. One approach is to implement low-level control (PWM, motor control, safety features, etc.) in dedicated components that are decoupled from high-level control (position, velocity, torque, etc.). In many cases, this can enable users to leverage common consumer technologies that can significantly reduce development time.
  +Over time, we have come to believe that robotics research can be greatly accelerated by designing systems that relax hard real-time requirements on the software that is exposed to users. One approach is to implement low-level control (motor control and safety features) in dedicated components that are decoupled from high-level control (position, velocity, torque, etc.). In many cases, this can enable users to leverage common consumer hardware and software tools that can accelerate development.
   
  -For example, the Biorobotics lab at CMU has always built a lot of custom hardware, so many of the people there have a mechanical or electrical engineering background. They tend to be most comfortable in Windows or OSX, and MATLAB is often the only programming language that they are familiar with. In the past, a limitation to support only C++ and Linux has posed a significant barrier to doing research. After we started to add cross-platform support and bindings for MATLAB (in ~2011), we ended up seeing a significant increase in our research output that roughly doubled the lab's paper publications related to snake robots. In particular, the lab has been able to develop and demonstrate new complex behaviors that would've been difficult to prototype beforehand (see https://youtu.be/NJ1FIsjt0yE[compliant control] or https://youtu.be/0CNQMiQnesc[inside pipes]).
  -
  -Over a series of blog posts, I'll try to share some of my own impressions and findings that have stemmed from the last several years spent on creating tools for robotics research. Today's post will focus on operating systems and measuring latency.
  +For example, the Biorobotics lab at Carnegie Mellon has researchers that tend to be mechanical or electrical engineers, rather than computer scientists or software engineers. As such tend they to be less familiar with Linux and C/C++ and much more comfortable with Windows/macOS and scripting languages like MATLAB. After our lab started providing cross-platform support and bindings for MATLAB (in ~2011), we ended up seeing a significant increase in our research output that roughly doubled the lab's paper publications related to snake robots. In particular, the lab has been able to develop and demonstrate new complex behaviors that would have been difficult to prototype beforehand (see https://youtu.be/NJ1FIsjt0yE[compliant control] or https://youtu.be/0CNQMiQnesc[inside pipes]).
   
   == Measuring Latency
   
  -Robots are controlled in _real-time_, which means that a command gets executed within a _deadline_ (fixed period of time). There are _hard real-time_ systems that must never exceed their deadline, and _soft real-time_ systems that are able to occasionally handle reasonable outliers. Missing deadlines when performing motor control of a robot can result in unwanted motions and 'jerky' behavior. 
  +Robots are controlled in _real-time_, which means that a command gets executed within a _deadline_ (fixed period of time). There are _hard real-time_ systems that must never exceed their deadline, and _soft real-time_ systems that are able to occasionally exceed their deadline. Missing deadlines when performing motor control of a robot can result in unwanted motions and 'jerky' behavior. 
   
  -Although there is a lot of information on the theoretical definition of these terms, it can be challenging to determine the maximum deadline (point at which a system's performance starts to degrade) for practical applications. This is especially true for research institutions that build unique mechanisms. Many research groups end up assuming that everything needs to be hard real-time with very stringent deadlines. While this is safe, it can cause a lot of unnecessary development efforts.
  +Although there is a lot of information on the theoretical definition of these terms, it can be challenging to determine the maximum deadline (point at which a system's performance starts to degrade or become unsafe) for practical applications. This is especially true for research institutions that build novel mechanisms and target cutting-edge applications. Many research groups end up assuming that everything needs to be hard real-time with very stringent deadlines. While this approach provides solid performance guarantees, it can also create a lot of unnecessary development overhead.
   
   // Data is not normally distributed. What is a better way to look at latency? What are tools that do this? How does jHiccup work? Gil Tene mentions coordinated omission, but that is less of a problem for request/response systems.
   
   //The first important realization when looking at latency is that data does not follow a Gaussian distribution. I have seen many data sets where the worst observed case was more than 1000 standard deviations away from the mean. Looking at only the mean and standard deviation tends to provide an extremely optimistic view that can be misleading.
   
  -Many benchmarks and tools make the assumption that latency follows a Gaussian distribution and report only the mean and the standard deviation. Unfortunately, latency is typically multi-modal and the most important part of the distribution when it comes to determinism are the 'outliers'. Even if a system's latency behaves as expected in 99% of the cases, the leftover 1% can be worse than all of the other 99% of measurements combined. Looking at only the mean and standard deviation completely fails to capture the more systemic issues. For example, I've seen many data sets where the worst observed case was more than 1000 standard deviations away from the mean. Such stutters are usually the main problem when working on real robotic systems.
  +Many benchmarks and tools make the assumption that latency follows a https://en.wikipedia.org/wiki/Gaussian_function[Gaussian distribution] and report only the mean and the standard deviation. Unfortunately, latency tends to be very https://en.wikipedia.org/wiki/Multimodal_distribution[multi-modal] and the most important part of the distribution when it comes to determinism are the 'outliers'. Even if a system's latency behaves as expected in 99% of the cases, the leftover 1% can be worse than all of the other 99% of measurements combined. Looking at only the mean and standard deviation completely fails to capture the more systemic issues. For example, I've seen many data sets where the worst observed case was more than 1000 standard deviations away from the mean. Such stutters are usually the main problem when working on real robotic systems.
   
  -Because of this, a more appropriate way to look at latency is via histograms and percentile plots, e.g., "_99.9% of measurements were below X ms_". There are several good resources about recording latency out there that I recommend checking out, such as link:https://youtu.be/lJ8ydIuPFeU[How NOT to Measure Latency] or link:http://psy-lob-saw.blogspot.com/2015/02/hdrhistogram-better-latency-capture.html[HdrHistogram: A better latency capture method].
  +Because of this, a more appropriate way to look at latency is via histograms and percentile plots, e.g., _"99.9% of measurements were below X ms"_. There are several good resources about recording latency out there that I recommend checking out, such as https://youtu.be/lJ8ydIuPFeU[How NOT to Measure Latency] or http://psy-lob-saw.blogspot.com/2015/02/hdrhistogram-better-latency-capture.html[HdrHistogram: A better latency capture method].
   
   == Operating Systems
   
  
positive.code:  
  The operating system is at the base of everything. No matter how performant the high-level software stack is, the system is fundamentally bound by the capabilities of the OS, it's scheduler, and the overall load on the system. Before you start optimizing your own software, you should make sure that your goal is actually achievable on the underlying platform.
  
  There is a trade-off between responding in a timely manner and overall performance, battery life, as well as several other concerns. As a result, the major consumer operating systems don't guarantee to meet hard deadlines and can theoretically have arbitrarily long pauses. 
  
  However, since using operating systems that users are familiar with can significantly ease development, it is worth evaluating their actual performance and capabilities. Even though there may not be any theoretical guarantees, the practical differences are often not noticeable.
  
  
negative.code: 
  //Dave: I might move the following paragraph up to the intro.  And add a sentence saying that what we advocate is finding ways to soften the requirements of hard-realtime comms for robotic systems.  For us a big part of it is engineering a modular low-level block that does RT control, but we still care a lot about understanding and measuring latency details at the high level.
  
  We've had very good experiences with implementing the low-level control (PID loops, motor control, safety features, etc.) on a per actuator level. That way all of the safety critical and latency sensitive pieces get handled by a dedicated RTOS and are independent of user code. The high-level controller (trajectories and multi-joint coordination) then only needs to update set targets (e.g. position/velocity/torque), which is far less sensitive to latency and doesn't require hard real-time communications. This approach enables quick prototyping of high-level behaviors using 'non-deterministic' technologies, such as Windows, MATLAB and standard UDP messages.
  
  For example, the high-level control in https://youtu.be/zaPtxre4tFc[Teleop Taxi] was done over Wi-Fi from MATLAB running on Windows, while simultaneously streaming video from an Android phone in the back of the robot. By removing the requirement for a local control computer, it only took 20-30 lines of code (see  https://gist.github.com/ennerf/b349c56d320da1db89b298fd807f00e4[simplified], https://gist.github.com/ennerf/7d59a9765da25ed7c02117da1805551c[full]) to run the entire demo. Actually using a local computer resulted in no perceivable benefit. While not every system can be controlled entirely through Wi-Fi, we've seen similar results even with more complex systems.
  
  