anchor.diff:  ==	Final Notes
   
   //It's easy to do 100 Hz control in just about any OS. 1KHz hard real-time requires lots of tuning. Rates are highly dependent on the application. Hard to generalize.
  +I've tried to provide a basic idea of the out of the box performance of various off the shelf operating systems. Actual requirements are heavily dependent on the specific use case, so it's impossible to make a general recommendation. While some safety critical applications may require a real time operating system (RTOS), others may run perfectly fine on commodity systems. My goal is to provide an overview that allows you to make a more informed decision about suitable platform after you've established your requirements. I'm planning on doing a future post on actually figuring out sane requirements.
   
  -I've tried to provide a basic idea of the out of box performance of various off the shelf operating systems. It's almost impossible to make any recommendations because the requirements are so specific to the use case. While some safety critical applications may require a real time operating system, others may run perfectly fine on Windows. My goal is to provide an overview that allows you to select a suitable choice after you've established your requirements. I'm planning on doing a future post on actually figuring out sane requirements.
  -
  -Note that these are not your only options. There are many different link:https://en.wikipedia.org/wiki/Comparison_of_real-time_operating_systems[RTOS] out there. There are even RT extensions for Windows, e.g.,  link:http://www.tenasys.com/overview-ifw[INtime] or link:http://kithara.com/en/products/realtime-suite[Kithara]. However, since integrating such systems can be very expensive or time consuming, I'd recommend going with simple and community supported solutions unless necessary.
  +Note that these are not your only options. There are many different link:https://en.wikipedia.org/wiki/Comparison_of_real-time_operating_systems[RTOS] out there. There are even real-time extensions for Windows, e.g., link:http://www.tenasys.com/overview-ifw[INtime] or link:http://kithara.com/en/products/realtime-suite[Kithara]. However, since integrating such systems can be very expensive or time consuming, I'd recommend going with simple and community supported solutions unless necessary.
   
   === Latency is not Gaussian
   
  
positive.code: 
  The first important realization when looking at latency is that data does not follow a Gaussian distribution. I have seen many data sets where the worst observed case was more than 1000 standard deviations away from the mean. Looking at only the mean and standard deviation tends to provide an extremely optimistic view that can be misleading.
  
  A better way to look at latency is via histograms and percentile plots, e.g., 99.9% of measurements have been below Xms. There are already several good resources about recording latency out there, so I won't go into detail here. Please refer to link:http://psy-lob-saw.blogspot.com/2015/02/hdrhistogram-better-latency-capture.html[HdrHistogram: A better latency capture method] and link:https://youtu.be/lJ8ydIuPFeU[How NOT to Measure Latency] for more information.
  
  //link:http://latencytipoftheday.blogspot.com/[Gil Tene]
  //video::lJ8ydIuPFeU[youtube]
  
  == Operating Systems
  
  //jHiccup is a great tool developed by Azul Systems that allows us to measure and record hiccups ('jitter')  at the OS level. These can be caused by a large number of reasons, including swap, indexing tasks, and many more. By running it on an idle system, we can measure the best case scenario.
   
  The operating system is the base of everything. No matter how amazing the software stack is, the system is fundamentally bound by the capabilities of the OS, it's scheduler, and the overall load on the system.
   
  There is a trade-off between responding in a timely manner and overall performance, battery life, and many other concerns. As a result, the major consumer operating systems don't guarantee to meet hard deadlines and can theoretically have arbitrarily long pauses.
  
  However, before dismissing consumer OS for real-time control systems, it is worth looking at their actual performance. Even though there are no theoretical guarantees, the practical performance may be good enough for many applications.
  
  === Benchmark Setup
  
  link:https://www.azul.com[Azul Systems] sells products targeted at latency sensitive applications and they have created a variety of useful tools to measure latency. link:https://www.azul.com/jhiccup/[jHiccup] is a tool that allows us to measure and record hiccups (~jitter) at the OS level. By running it on an idle system, we can measure the best case scenario. It uses link:https://github.com/HdrHistogram/HdrHistogram[HdrHistogram] for the actual recording of samples. Lastly, I'm using link:https://github.com/ennerf/HdrHistogramVisualizer[HdrHistogramVisualizer] which is a tool that I've written for visualizing the recorded data.
  
  Hiccups can be caused by a large number of reasons, including scheduling, paging, indexing, and many more. jHiccup measures the time for sleep(1ms) and records the delta to the fastest previously recorded sample. For example, if the fastest sample was 1ms, but it took 3ms to wake up, it will record a 2ms hiccup value.
  
  I've setup two standard desktop computers, one for Windows/Linux and one for Mac tests.
  
  
negative.code: # set priority
  sudo chrt -p 98 $pid
  ---------------------------------------------------------------------
  
  Below is a comparison of the two Linux variants. 24 hours with 3 minute intervals,
  
  