anchor.diff:  Most video streams that are sent over the network get compressed in order to save bandwidth (exceptions include GigE cameras). While the cameras often include specialized hardware (link:https://en.wikipedia.org/wiki/Application-specific_integrated_circuit[ASIC]) to do the encoding, the decoding on the computer side tends to happen in Software. Since this can take a significant amount of time, it is common practice to move the acquisition to a separate thread in order to reduce the load on the main processing thread.
   
   [NOTE]
  -Note that Intel has added hardware support for encoding and decoding H.264 to their CPUs starting in 2011 (link:https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video[QuickSync]), but there is unfortunately a lack of library support.
  -
  -Due to MATLAB's single threaded nature, we needed to resort to external languages for getting background threads. Out of the box MATLAB provided two interfaces to other languages:  https://www.mathworks.com/help/matlab/matlab_external/introducing-mex-files.html[MEX] for calling C/C++ code, and the https://www.mathworks.com/help/matlab/matlab_external/product-overview.html[Java Interface] for calling Java code. For a variety of reasons, such as simplified development and cross-platform deployment, we decided to use the Java interface. The diagram below shows a simplified overview of the resulting system.
  +Note that Intel has added hardware support for encoding and decoding H.264 to their CPUs starting in 2011 (link:https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video[QuickSync]), but there is unfortunately a lack of cross-platform library support.
  +
  +Due to MATLAB's single threaded nature, we needed to resort to external languages for getting background threads. Out of the box MATLAB provided two interfaces to other languages:  https://www.mathworks.com/help/matlab/matlab_external/introducing-mex-files.html[MEX] for calling C/C++ code, and the https://www.mathworks.com/help/matlab/matlab_external/product-overview.html[Java Interface] for calling Java code.
  +
  +Even our internal projects need to support all major operating systems, so we tend to favor the Java interface for its built-in cross-platform support. Java libraries are also simpler to develop and deploy. However, while the internal performance differences between Java and C++ rarely matter, the overhead of calling Java methods from MATLAB is significantly higher than the overhead of calling MEX functions. This becomes especially noticeable when translating large amounts of data such as high resolution images. 
  +
  +We still decided on first trying the Java interface, but we weren't entirely certain whether it'd be feasible. The diagram below shows a simplified view of the resulting system.
   
   [.text-center]
   .System overview for a stereo vision setup
   image::{imagesdir}/streaming/stereocam-matlab.svg[width=100%]
   
  -Starting background threads and getting the video data into Java was comparatively straight forward. We used the excellent link:https://github.com/bytedeco/javacv[JavaCV] library, which is a Java wrapper around link:https://opencv.org/[OpenCV] and link:https://www.ffmpeg.org/[FFMpeg] that includes pre-compiled native binaries for all major operating systems. The main challenge was to get the image data across the translation layer back into MATLAB.
  -
  -In order to process images at a frame rate of 30 fps in real-time, the total time budget of the main MATLAB thread is 33ms per cycle. Thus, the aquisition overhead imposed on the main thread needs to be sufficiently low, i.e., a low number of milliseconds, to leave enough time for the actual processing.
  -
  -While the automated translation from and to Java types tends to be sufficient for most applications, it is very inefficient for large and multi dimensional matrices. For example, a `1080x1920x3` MATLAB image matrix gets translated to a `byte[1080][1920][3]` Java array, which means that every single pixel in the image gets stored in a separate array object.
  -
  -As an additional complication, Java uses a different memory layout than MATLAB. While Java stores image data in row-major order, MATLAB stores images transposed and in column-major order. For example, if the pixel data for a Java image would be laid out as `[RGB][RGB][RGB]...`, the same image would be laid out as `[RRR...][GGG...][BBB...]` in MATLAB.  (`R`, `G`, `B` correspond to red, green, and blue values for one pixel)
  +Starting background threads and getting the video data into Java was relatively straight forward. We used the excellent link:https://github.com/bytedeco/javacv[JavaCV] library, which is a Java wrapper around link:https://opencv.org/[OpenCV] and link:https://www.ffmpeg.org/[FFMpeg] that includes pre-compiled native binaries for all major operating systems.
  +
  +As expected, the main challenge was getting the image data across the translation layer into MATLAB. In order to process images at a frame rate of 30 fps in real-time, the total time budget of the main MATLAB thread is 33ms per cycle. Thus, the aquisition overhead imposed on the main thread needs to be sufficiently low, i.e., a low number of milliseconds, to leave enough time for the actual processing.
  +
  +While the automated translation from and to Java types tends to be sufficient for most applications, it is very inefficient for large and multi dimensional matrices. For example, a `1080x1920x3` MATLAB image matrix gets translated to a `byte[1080][1920][3]` Java array, which means that every single pixel in the image gets stored in a separate array.
  +
  +As an additional complication, Java uses a different memory layout than MATLAB. While Java stores image data in row-major order, MATLAB stores the pixels transposed and in column-major order. For example, if the pixels for a Java image would be laid out as `[RGB][RGB][RGB]...`, the same image would be laid out as `[RRR...][GGG...][BBB...]` in MATLAB.  (`R`, `G`, `B` correspond to red, green, and blue values for one pixel)
   
   ////
   * Acquiring video on a background thread
  
positive.code: 
  == Data Translation
  
  We compared five different ways to get image data from Java into MATLAB. Note that the benchmarks in this section measure only the overhead on the MATLAB main thread. The reason for this is that while additional overhead in a background thread may increase overall latency, it doesn't impact the time budget available for the processing. The full benchmark code is available link:https://github.com/HebiRobotics/HebiCam/tree/benchmark[here].
  
  *1. Default 3D Array*
  
  
negative.code: 
  *2. Compressed 1D Array*
  
  A common approach to move image data across distributed components (e.g. link:http://www.ros.org/[ROS]) is to encode the individual images using link:https://en.wikipedia.org/wiki/Motion_JPEG[MJPEG] compression before sending them. Doing this within a single process is obviously wasteful, but we included it because it is common practice in some systems. Since MATLAB does not offer a way to decompress jpeg images in memory, we needed to save to data to file (located on a RAM disk) first.
  
  [source,matlab]
  ----
  