anchor.diff:  
   == Motivation
   
  -For our booth at ICRA 2014 we put together a demo system in MATLAB that used stereo vision for tracking colored bean bags, and a robot arm to pick them up. We used two IP cameras that streamed link:https://de.wikipedia.org/wiki/H.264[H.264] video over link:https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol[RTSP]. While developing the image processing and robot control parts were comparatively simple, it proved to be a challenge to get streaming video data from both cameras into MATLAB fast enough to be useful.
  -
  -* link:http://www.mathworks.com/hardware-support/ip-camera.html[IP Camera Support] didn't exist at the time and only supported link:https://en.wikipedia.org/wiki/Motion_JPEG[MJPEG] over link:https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol[HTTP]
  -* link:http://www.mathworks.com/hardware-support/matlab-webcam.html[USB Webcam Support] only supported USB cameras
  -* link:http://www.mathworks.com/help/matlab/ref/imread.html[imread] and link:http://www.mathworks.com/help/matlab/ref/webread.html[webread] were limited to HTTP and too slow for real-time
  +For our booth at ICRA 2014 we put together a demo system in MATLAB that used stereo vision for tracking colored bean bags, and a robot arm to pick them up. We used two IP cameras that streamed link:https://de.wikipedia.org/wiki/H.264[H.264] video over link:https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol[RTSP]. While developing the image processing and robot control parts worked as expected, it proved to be a challenge to get streaming video data from two cameras into MATLAB fast enough to be useful.
  +
  +* link:http://www.mathworks.com/hardware-support/ip-camera.html[IP Camera Support] didn't exist at the time and only supports link:https://en.wikipedia.org/wiki/Motion_JPEG[MJPEG] over link:https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol[HTTP]
  +* link:http://www.mathworks.com/hardware-support/matlab-webcam.html[USB Webcam Support] only supports USB cameras
  +* link:http://www.mathworks.com/help/matlab/ref/imread.html[imread] and link:http://www.mathworks.com/help/matlab/ref/webread.html[webread] are limited to HTTP and too slow for real-time
   
   Since we didn't want to port the code to another language, we instead wrote a small library for acquiring streaming video. The project was later open sourced as link:http://www.github.com/HebiRobotics/HebiCam[HebiCam].
   
  
positive.code: Most video streams that are sent over the network get compressed in order to save bandwidth (exceptions include GigE cameras). While the cameras often include specialized hardware (link:https://en.wikipedia.org/wiki/Application-specific_integrated_circuit[ASIC]) to do the encoding, the decoding on the computer side tends to happen in Software. Since this can take a significant amount of time, it is common practice to move the acquisition to a separate thread in order to reduce the load on the main processing thread.
  
  [NOTE]
  Note that Intel has added hardware support for encoding and decoding H.264 to their CPUs starting in 2011 (link:https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video[QuickSync]), but there is unfortunately a lack of library support.
  
  Due to MATLAB's single threaded nature, we needed to resort to external languages for getting background threads. Out of the box MATLAB provided two interfaces to other languages:  https://www.mathworks.com/help/matlab/matlab_external/introducing-mex-files.html[MEX] for calling C/C++ code, and the https://www.mathworks.com/help/matlab/matlab_external/product-overview.html[Java Interface] for calling Java code. For a variety of reasons, such as simplified development and cross-platform deployment, we decided to use the Java interface. The diagram below shows a simplified overview of the resulting system.
  
  [.text-center]
  .System overview for a stereo vision setup
  image::{imagesdir}/streaming/stereocam-matlab.svg[width=100%]
  
  Starting background threads and getting the video data into Java was comparatively straight forward. We used the excellent link:https://github.com/bytedeco/javacv[JavaCV] library, which is a Java wrapper around link:https://opencv.org/[OpenCV] and link:https://www.ffmpeg.org/[FFMpeg] that includes pre-compiled native binaries for all major operating systems. The main challenge was to get the image data across the translation layer back into MATLAB.
  
  In order to process images at a frame rate of 30 fps in real-time, the total time budget of the main MATLAB thread is 33ms per cycle. Thus, the aquisition overhead imposed on the main thread needs to be sufficiently low, i.e., a low number of milliseconds, to leave enough time for the actual processing.
  
  While the automated translation from and to Java types tends to be sufficient for most applications, it is very inefficient for large and multi dimensional matrices. For example, a `1080x1920x3` MATLAB image matrix gets translated to a `byte[1080][1920][3]` Java array, which means that every single pixel in the image gets stored in a separate array object.
  
  As an additional complication, Java uses a different memory layout than MATLAB. While Java stores image data in row-major order, MATLAB stores images transposed and in column-major order. For example, if the pixel data for a Java image would be laid out as `[RGB][RGB][RGB]...`, the same image would be laid out as `[RRR...][GGG...][BBB...]` in MATLAB.  (`R`, `G`, `B` correspond to red, green, and blue values for one pixel)
  
  ////
  * Acquiring video on a background thread
  
negative.code: .Conversion overhead on the MATLAB thread in [ms]
  image::{imagesdir}/streaming/table_performance.svg[width=100%]
  
  The results show that the default conversion as well as jpeg compression are essentially non-starters for color images. For grayscale images the default conversion works significantly better due to the fact that the data is stored in a much more efficient 2D array (`byte[height][width]`).
  
  [NOTE]
  Note that we currently don't have a good explanation for the ~10x cost increase (rather than ~4x) between 1080p and 4K grayscale. The behavior was the same across computers and different memory settings.
  
  When copying the pixels of a Java image we can see another significant performance increase due to the data being stored in a single contiguous array. At this point much of the overhead comes from re-ordering pixels, so by doing the conversion beforehand we can get another 2-3x improvement.
  
  Lastly, although accessing shared memory in combination with the locking overhead results in a slightly higher fixed cost, the copying itself is significantly cheaper, resulting in another 2-3x speedup for high resolution images. Overall, the approach scales very well and would even allow streaming of 4K color images from two cameras simultaneously.
  
  == Final Notes
  
  As for our initial motiation, after creating link:https://github.com/HebiRobotics/HebiCam[HebiCam] we were able to develop and reliably run the whole demo entirely in MATLAB.
  
  One lesson learned was that although MATLAB's Java interface can be inefficient for certain cases, there are simple workarounds that can remove most bottlenecks. The most important rule is to avoid converting large multi-dimensional arrays whenever possible.
  
  Another lesson learned was that shared memory provides a very efficient way to transfer large amounts of data to and from MATLAB. We also found it useful for inter-process communication between multiple MATLAB instances, e.g., one instance can track a target while another instance can use its output for real-time control. This can be useful to avoid coupling a fast control loop to the (lower) frame rate of a camera or sensor.
  
  ////
  The biggest takeaway for us was that MATLAB supports shared memory. We had used shared memory before for inter-process communication in Java and C++, but we somehow assumed that MATLAB wouldn't support it. We eventually looked into this and found that MATLAB does have support, and that the implementation is actually quite nice. The data definition can even map the memory to an array of structs with multiple variables.
  