anchor.diff:  
   Zoomed in we can see that the Windows hiccups are actually very repeatable. 99.9% are below 2ms, but there are frequent spikes to a single value at around 16ms. This also highlights the importance of looking at the details of the latency distribution. In other data sets that I've seen, it is rare for the worst case to be equal to the 99.99% percentile. It's also interesting that the distribution for 10 minutes looks identical to the 24 hour chart. OSX shows similar behavior, but with lower spikes. Ubuntu 16.04 is overall very quiet.
   
  -It's debatable whether this makes any difference for robotic systems in practice. All of the systems I've worked with either had hard real-time requirements below 1ms, in which case none of these OS would be sufficient, or they were soft real-time systems that could handle occasional hiccups to 25 or even 100 ms. I have yet to see a robotic system that performed perceivably worse on Windows versus Linux.
  +It's debatable whether this makes any difference for robotic systems in practice. All of the systems I've worked with either had hard real-time requirements below 1ms, in which case none of these OS would be sufficient, or they were soft real-time systems that could handle occasional hiccups to 25 or even 100 ms. I have yet to see one of our robotic systems perform perceivably worse on Windows versus Linux.
   
   === Real Time Linux
   
  
positive.code: .Linux vs. RT Linux hiccups (10 min)
  image::os/ubuntu-scl_10m.png[Linux vs. RT Linux hiccups (10 min), link="/images/os/ubuntu-scl_10m.png"]
  
  I've also added the 24 hour chart for the real-time variant by itself to provide a better scale. Note that this resolution is getting close to what we can measure and record.
  
  .RT Linux hiccups (24h)
  image::os/scl_24h.png[RT Linux hiccups (24h), link="/images/os/scl_24h.png"]
  
negative.code: 
  //The first important realization when looking at latency is that data does not follow a Gaussian distribution. I have seen many data sets where the worst observed case was more than 1000 standard deviations away from the mean. Looking at only the mean and standard deviation tends to provide an extremely optimistic view that can be misleading.
  
  Many benchmarks and tools make the assumption that latency follows a Gaussian distribution and report only the mean and the standard deviation. Unfortunately, latency is typically multi-modal and the most important part of the distribution when it comes to determinism are the 'outliers'. Even if a system's latency behaves as expected in 99% of the cases, the leftover 1% can be worse than all of the other 99% of measurements combined. Looking at only the mean and standard deviation completely fails to capture the more systemic issues. For example, I've seen many data sets where the worst observed case was more than 1000 standard deviations away from the mean. Such stutters are usually the main problem when working on real robotic systems.
  
  Because of this, a more appropriate way to look at latency is via histograms and percentile plots, e.g., "_99.9% of measurements were below X ms_". There are several good resources about recording latency out there that I recommend checking out, such as link:https://youtu.be/lJ8ydIuPFeU[How NOT to Measure Latency] or link:http://psy-lob-saw.blogspot.com/2015/02/hdrhistogram-better-latency-capture.html[HdrHistogram: A better latency capture method].
  
  == Operating Systems
  
  