anchor.diff:  
   *2. Compressed 1D Array*
   
  -A common approach to move image data across distributed components (e.g. link:http://www.ros.org/[ROS]) is to encode the individual images using link:https://en.wikipedia.org/wiki/Motion_JPEG[MJPEG] compression before sending them. Doing this within a single process is obviously wasteful, but we included it because it is common practice in some systems. Since MATLAB does not offer a way to decompress jpeg images in memory, we needed to save to data to file (located on a RAM disk) first.
  +A common approach to move image data across distributed components (e.g. link:http://www.ros.org/[ROS]) is to encode the individual images using link:https://en.wikipedia.org/wiki/Motion_JPEG[MJPEG] compression. Doing this within a single process is obviously wasteful, but we included it because it is common practice in many distributed systems. Since MATLAB did not offer a way to decompress jpeg images in memory, we needed to save the compressed data to a file located on a RAM disk.
   
   [source,matlab]
   ----
  
positive.code: image = reshape(vector,h,w,c);
  ----
  
  The most efficient way we found to convert the memory layout on the Java side was to use OpenCV's `split` and `transpose` functions. The code can be found in link:https://github.com/HebiRobotics/HebiCam/blob/master/src/main/java/us/hebi/matlab/streaming/MatlabImageConverterBGR.java[MatlabImageConverterBGR] and link:https://github.com/HebiRobotics/HebiCam/blob/master/src/main/java/us/hebi/matlab/streaming/MatlabImageConverterGrayscale.java[MatlabImageConverterGrayscale].
  
  *5. MATLAB Layout as Shared Memory*
  
  This approach is the same as 4. with the difference that the Java translation layer is bypassed entirely by using shared memory via link:https://mathworks.com/help/matlab/ref/memmapfile.html[memmapfile]. Shared memory is typically used for inter-process communication, but it can also be used within a single process. Running within the same process also simplifies synchronization since MATLAB can access Java locks.
  
  [source,matlab]
  ----
  
negative.code: 
  == Motivation
  
  For our booth at ICRA 2014 we put together a demo system in MATLAB that used stereo vision for tracking colored bean bags, and a robot arm to pick them up. We used two IP cameras that streamed link:https://de.wikipedia.org/wiki/H.264[H.264] video over link:https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol[RTSP]. While developing the image processing and robot control parts worked as expected, it proved to be a challenge to acquire images from both video streams fast enough to be useful.
  
  * link:http://www.mathworks.com/hardware-support/ip-camera.html[IP Camera Support] only supports link:https://en.wikipedia.org/wiki/Motion_JPEG[MJPEG] over link:https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol[HTTP] and didn't exist at the time
  * link:http://www.mathworks.com/hardware-support/matlab-webcam.html[USB Webcam Support] only supports USB cameras
  * link:http://www.mathworks.com/help/matlab/ref/imread.html[imread] and link:http://www.mathworks.com/help/matlab/ref/webread.html[webread] are limited to HTTP and too slow for real-time
  
  Since we did not want to switch to another language, we decided to develop a small library for acquiring video streams. The project was later open sourced as link:http://www.github.com/HebiRobotics/HebiCam[HebiCam].
  
  == Technical Background
  
  In order to save bandwidth most IP cameras compress video before sending it over the network. Since the resulting decoding step can be computationally expensive, it is common practice to move the acquisition to a separate thread in order to reduce the load on the main processing thread.
  
  Doing this in MATLAB unfortunately requires some workarounds due to the language's single threaded nature, i.e., background threads need to run in another language. Out of the box there are two supported interfaces: https://www.mathworks.com/help/matlab/matlab_external/introducing-mex-files.html[MEX] for calling C/C++ code, and the https://www.mathworks.com/help/matlab/matlab_external/product-overview.html[Java Interface] for calling Java code.
  
  While both interfaces have strengths and weaknesses, either one can be used for solving practically any use case. Since this project had to work on all major operating systens, we chose to use the Java interface for its built-in cross-platform support and simplified deployment. The diagram below shows an overview of the resulting system.
  
  ////
  Since our projects almost always have to work on all major platforms, we tend to favor the Java interface for its simplified deployment and  built-in cross-platform support. However, one of the weaknesses is that the overhead of calling Java methods from MATLAB is higher than the overhead of calling MEX functions. This difference is usually negligible, but it unfortunately becomes noticeable when translating large amounts of data such as high resolution images. 
  
  
  although the internal performance differences between Java and C++ rarely matter, the overhead of calling Java methods from MATLAB is higher than the overhead of calling MEX functions. This difference is usually negligible, but it unfortunately becomes noticeable when translating large amounts of data such as high resolution images.
  ////
  
  [.text-center]
  .System overview for a stereo vision setup
  image::{imagesdir}/streaming/stereocam-matlab.svg[width=100%]
  
  Starting background threads and getting the video stream into Java was relatively straight forward. We used the link:https://github.com/bytedeco/javacv[JavaCV] library, which is a Java wrapper around link:https://opencv.org/[OpenCV] and link:https://www.ffmpeg.org/[FFMpeg] that includes pre-compiled native binaries for all major operating systems. However, passing the acquired image data from Java into MATLAB turned out to be more challenging.
  
  The Java interface automatically converts between Java and MATLAB types by following a set of link:https://www.mathworks.com/help/matlab/matlab_external/handling-data-returned-from-java-methods.html[rules]. This makes it much simpler to develop for than the MEX interface, but it does cause additional overhead when calling Java functions. Most of the time this overhead is negligible. However, for certain types of data, such as large and multi-dimensional matrices, the default rules are very inefficient and can become prohibitively expensive. For example, a `1080x1920x3` MATLAB image matrix gets translated to a `byte[1080][1920][3]` Java array, which means that every single pixel in the image gets stored as a separate array.
  
  As an additional complication, Java images use a different memory layout than MATLAB image matrices. While Java stores pixels in row-major order, MATLAB stores them transposed and in column-major order. For example, if the pixels for a Java image would be laid out as `[RGB][RGB][RGB]...`, the same image would be laid out as `[RRR...][GGG...][BBB...]` in MATLAB.  (`R`, `G`, `B` correspond to red, green, and blue values for one pixel)
  
  In order to process images at a frame rate of 30 fps in real-time, the total time budget of the main MATLAB thread is 33ms per cycle. Thus, the aquisition overhead imposed on the main thread needs to be sufficiently low, i.e., a low number of milliseconds, to leave enough time for the actual processing.
  
  
  ////
  * Acquiring video on a background thread
  