anchor.diff:  <script>hljs.initHighlightingOnLoad()</script>
   ++++
   
  -The acquisition and processing of a video stream can be very computationally expensive. Typical image processing applications split the work across multiple threads, one acquiring the images, and another one running the actual algorithms. In MATLAB we can get multi-threading by interfacing with other languages, but there is a significant cost associated to exchanging data across the resulting language barrier. In this blog post we compare different approaches for getting data through MATLAB's Java interface, and we show how to acquire high resolution video streams in real-time and with low overhead.
  +The acquisition and processing of a video stream can be very computationally expensive. Typical image processing applications split the work across multiple threads, one acquiring the images, and another one running the actual algorithms. In MATLAB we can get multi-threading by interfacing with other languages, but there is a significant cost associated with exchanging data across the resulting language barrier. In this blog post, we compare different approaches for getting data through MATLAB's Java interface, and we show how to acquire high-resolution video streams in real-time and with low overhead.
   
   == Motivation
   
  -For our booth at ICRA 2014 we put together a demo system in MATLAB that used stereo vision for tracking colored bean bags, and a robot arm to pick them up. We used two IP cameras that streamed link:https://de.wikipedia.org/wiki/H.264[H.264] video over link:https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol[RTSP]. While developing the image processing and robot control parts worked as expected, it proved to be a challenge to acquire images from both video streams fast enough to be useful.
  +For our booth at ICRA 2014, we put together a demo system in MATLAB that used stereo vision for tracking colored bean bags, and a robot arm to pick them up. We used two IP cameras that streamed link:https://de.wikipedia.org/wiki/H.264[H.264] video over link:https://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol[RTSP]. While developing the image processing and robot control parts worked as expected, it proved to be a challenge to acquire images from both video streams fast enough to be useful.
   
   * link:http://www.mathworks.com/hardware-support/ip-camera.html[IP Camera Support] only supports link:https://en.wikipedia.org/wiki/Motion_JPEG[MJPEG] over link:https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol[HTTP] and didn't exist at the time
   * link:http://www.mathworks.com/hardware-support/matlab-webcam.html[USB Webcam Support] only supports USB cameras
  
positive.code: 
  In order to save bandwidth most IP cameras compress video before sending it over the network. Since the resulting decoding step can be computationally expensive, it is common practice to move the acquisition to a separate thread in order to reduce the load on the main processing thread.
  
  Doing this in MATLAB unfortunately requires some workarounds due to the language's single threaded nature, i.e., background threads need to run in another language. Out of the box there are two supported interfaces: https://www.mathworks.com/help/matlab/matlab_external/introducing-mex-files.html[MEX] for calling C/C++ code, and the https://www.mathworks.com/help/matlab/matlab_external/product-overview.html[Java Interface] for calling Java code.
  
  While both interfaces have strengths and weaknesses, practically all use cases can be solved using either one. For this project we chose to use the Java interface because we needed to support all major operating systems, and using Java greatly simplified cross-platform development as well as the deployment of binaries. The diagram below shows an overview of the resulting system.
  
  [.text-center]
  .System overview for a stereo vision setup
  image::{imagesdir}/streaming/stereocam-matlab.svg[width=100%]
  
  Starting background threads and getting the video stream into Java was relatively straight forward. We used the link:https://github.com/bytedeco/javacv[JavaCV] library, which is a Java wrapper around link:https://opencv.org/[OpenCV] and link:https://www.ffmpeg.org/[FFMpeg] that includes pre-compiled native binaries for all major platforms. However, passing the acquired image data from Java into MATLAB turned out to be more challenging.
  
  The Java interface automatically converts between Java and MATLAB types by following a set of link:https://www.mathworks.com/help/matlab/matlab_external/handling-data-returned-from-java-methods.html[rules]. This makes it much simpler to develop for than the MEX interface, but it does cause additional overhead when calling Java functions. Most of the time this overhead is negligible. However, for certain types of data, such as large and multi-dimensional matrices, the default rules are very inefficient and can become prohibitively expensive. For example, a `1080x1920x3` MATLAB image matrix gets translated to a `byte[1080][1920][3]` in Java, which means that there is a separate array object for every single pixel in the image.
  
  As an additional complication, MATLAB stores image data in a different memory layout than most other libraries (e.g. OpenCV's `Mat` or Java's `BufferedImage`). While pixels are commonly stored in row-major order (`[width][height][channels]`), MATLAB stores images transposed and in column-major order (`[channels][width][height]`). For example, if the Red-Green-Blue pixels of a `BufferedImage` would be laid out as `[RGB][RGB][RGB]...`, the same image would be laid out as `[RRR...][GGG...][BBB...]` in MATLAB. Depending on the resolution this conversion can become fairly expensive.
  
  In order to process images at a frame rate of 30 fps in real-time, the total time budget of the main MATLAB thread is 33ms per cycle. Thus, the aquisition overhead imposed on the main thread needs to be sufficiently low, i.e., a low number of milliseconds, to leave enough time for the actual processing.
  
  == Data Translation
  
  
negative.code: 
  *5. MATLAB Layout as Shared Memory*
  
  The fifth approach is the same as the fourth with the difference that the Java translation layer is bypassed entirely by using shared memory via `link:https://mathworks.com/help/matlab/ref/memmapfile.html[memmapfile]`. Shared memory is typically used for inter-process communication, but it can also be used within a single process. Running within the same process also simplifies synchronization since MATLAB can access Java locks.
  
  [source,matlab]
  ----
  