anchor.diff:  <script>hljs.initHighlightingOnLoad()</script>
   ++++
   
  -The acquisition and processing of a video stream can be very computationally expensive. Typical image processing applications split the work across multiple threads, one acquiring the images, and another one running the actual algorithms. In MATLAB we can get multi-threading by interfacing with other languages, but there is a significant cost associated to exchanging data across the resulting language barrier. In this blog post we compare different approaches for getting data through MATLAB's Java interface, and we show how to acquire high resolution video streams in real-time and with low overhead.
  +The acquisition and processing of a video stream can be very computationally expensive. Typical image processing applications split the work across multiple threads, one acquiring the images, and another one running the actual algorithms. In MATLAB we can get multi-threading by interfacing with other languages, but there is a significant cost associated with exchanging data across the resulting language barrier. In this blog post, we compare different approaches for getting data through MATLAB's Java interface, and we show how to acquire high-resolution video streams in real-time and with low overhead.
   
   == Motivation
   
  -For our booth at ICRA 2014 we put together a demo system in MATLAB that used stereo vision for tracking colored bean bags, and a robot arm to pick them up. We used two IP cameras that streamed link:https://de.wikipedia.org/wiki/H.264[H.264] video over link:https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol[RTSP]. While developing the image processing and robot control parts worked as expected, it proved to be a challenge to acquire images from both video streams fast enough to be useful.
  +For our booth at ICRA 2014, we put together a demo system in MATLAB that used stereo vision for tracking colored bean bags, and a robot arm to pick them up. We used two IP cameras that streamed link:https://de.wikipedia.org/wiki/H.264[H.264] video over link:https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol[RTSP]. While developing the image processing and robot control parts worked as expected, it proved to be a challenge to acquire images from both video streams fast enough to be useful.
   
   * link:http://www.mathworks.com/hardware-support/ip-camera.html[IP Camera Support] only supports link:https://en.wikipedia.org/wiki/Motion_JPEG[MJPEG] over link:https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol[HTTP] and didn't exist at the time
   * link:http://www.mathworks.com/hardware-support/matlab-webcam.html[USB Webcam Support] only supports USB cameras
  
positive.code: 
  The multiplication by one forces a copy of the data. This is necessary because under-the-hood `memmapfile` only returns a reference to the underlying memory.
  
  ////
  We can solve this by doing one of the following
  
  * Make the internal logic smart enough to handle cases where users didn't call `unlock`
  * Use link:https://mathworks.com/help/matlab/ref/oncleanup.html[onCleanup] to guarantee unlocking, e.g., `c = onCleanup(@()unlock(this.javaObj))`
  * Create a custom class that locks in the constructor and unlocks in the destructor
  
  However, even if the `unlock` call is in a destructor (guaranteed to run when exiting scope), the Java logic still needs to handle cases where the code gets interrupted before the `lock` call.
  ////
  
  == Results
  
  All benchmarks were run in MATLAB 2017b on an link:https://www.intel.com/content/www/us/en/products/boards-kits/nuc/kits/nuc6i7kyk.html[Intel NUC6I7KYK]. The performance was measured using MATLAB's link:https://mathworks.com/help/matlab/ref/timeit.html[timeit] function. The background color of each cell in the result tables represents a rough classification of the overhead on the main MATLAB thread.
  
  {set:cellbgcolor:white}
  
  
negative.code: 
  The two tables below show the results for converting color (RGB) images as well as grayscale images. All measurements are in milliseconds.
  
  [.text-center]
  .Conversion overhead on the MATLAB thread in [ms]
  image::{imagesdir}/streaming/table_performance.svg[width=100%]
  
  The results show that the default conversion, as well as jpeg compression, are essentially non-starters for color images. For grayscale images, the default conversion works significantly better due to the fact that the data is stored in a much more efficient 2D array (`byte[height][width]`), and that there is no need to re-order pixels by color.
  
  [NOTE]
  Unfortunately, we currently don't have a good explanation for the ~10x cost increase (rather than ~4x) between 1080p and 4K grayscale. The behavior was the same across computers and different memory settings.
  
  When copying the backing array of a `BufferedImage` we can see another significant performance increase due to the data being stored in a single contiguous array. At this point much of the overhead comes from re-ordering pixels, so by doing the conversion beforehand, we can get another 2-3x improvement.
  
  Lastly, although accessing shared memory in combination with the locking overhead results in a slightly higher fixed cost, the copying itself is significantly cheaper, resulting in another 2-3x speedup for high-resolution images. Overall, going through shared memory scales very well and would even allow streaming of 4K color images from two cameras simultaneously.
  
  == Final Notes
  
  As for our initial motivation, after creating link:https://github.com/HebiRobotics/HebiCam[HebiCam] we were able to develop and reliably run the whole demo entirely in MATLAB.
  
  video::R0nQSxt8uic?rel=0[youtube]
  
  Our main takeaway was that although MATLAB's Java interface can be inefficient for certain cases, there are simple workarounds that can remove most bottlenecks. The most important rule is to avoid converting to and from large multi-dimensional matrices whenever possible.
  
  Another insight was that shared-memory provides a very efficient way to transfer large amounts of data to and from MATLAB. We also found it useful for inter-process communication between multiple MATLAB instances. For example, one instance can track a target while another instance can use its output for real-time control. This is useful for avoiding coupling a fast control loop to the (usually lower) frame rate of a camera or sensor.
  
  