title: Android convert ImageReader Image to YCbCr_420_SP (NV21) byte array using render script?
body: <p>I am currently working with <a href="https://github.com/bytedeco/javacv" rel="nofollow noreferrer">Javacv</a> which makes use of the <code>public void onPreviewFrame(byte[] data, Camera camera)</code> camera function.</p>
  
  <p>Since camera is deprecated, I have been looking into <a href="https://developer.android.com/reference/android/hardware/camera2/package-summary.html" rel="nofollow noreferrer">camera2</a> and <a href="https://developer.android.com/reference/android/media/projection/MediaProjection.html" rel="nofollow noreferrer">MediaProjection</a>. Both of these libraries make use of the <a href="https://developer.android.com/reference/android/media/ImageReader.html" rel="nofollow noreferrer">ImageReader class</a>.</p>
  
  <p>Currently I instantiate such an <code>ImageReader</code> with the following code:</p>
  
  <p><code>ImageReader.newInstance(DISPLAY_WIDTH, DISPLAY_HEIGHT, PixelFormat.RGBA_8888, 2);</code></p>
  
  <p>And attach an <code>OnImageAvailableListener</code> like this:</p>
  
  <pre><code> private final ImageReader.OnImageAvailableListener mOnImageAvailableListener
              = new ImageReader.OnImageAvailableListener() {
  
          @Override
          public void onImageAvailable(ImageReader reader) {
  
              mBackgroundHandler.post(new processImage(reader.acquireNextImage()));
          }
  </code></pre>
  
  <p>I have tried using the <code>RGBA_8888</code> format with Javacv as per this thread: <a href="https://github.com/bytedeco/javacv/issues/298" rel="nofollow noreferrer">https://github.com/bytedeco/javacv/issues/298</a> but that doesn't work for me.</p>
  
  <p>So instead I was thinking about using <a href="https://developer.android.com/guide/topics/renderscript/compute.html" rel="nofollow noreferrer">Render script</a> to convert these <a href="https://developer.android.com/reference/android/media/Image.html" rel="nofollow noreferrer">Image</a>s to <code>NV21(YUV_420_SP)</code> format (which is the default output of camera in the <code>onPreviewFrame</code> function) as that worked for me with the <code>camera</code> library.</p>
  
  <p>I have also read posts such as <a href="https://stackoverflow.com/questions/5960247/convert-bitmap-array-to-yuv-ycbcr-nv21">this one</a> and <a href="http://www.equasys.de/colorconversion.html" rel="nofollow noreferrer">this website</a> to do the conversion, but these didn't work for me and I fear they will be too slow.
  Furthermore, my knowledge of C is severely limited.
  Basically it looks like I want the reverse operation of <a href="https://developer.android.com/reference/android/renderscript/ScriptIntrinsicYuvToRGB.html" rel="nofollow noreferrer">https://developer.android.com/reference/android/renderscript/ScriptIntrinsicYuvToRGB.html</a></p>
  
  <p>So how can you go from an <code>Image</code> to a byte array that would match the output of the <code>onPreviewFrame</code> function i.e. <code>NV21(YUV_420_SP)</code> format? Preferably using Renderscript as it's faster.</p>
  
  <hr>
  
  <h1>Edit 1:</h1>
  
  <p>I have tried using <code>ImageFormat.YUV_420_888</code> but to no avail. I kept getting errors like <code>The producer output buffer format 0x1 doesn't match the ImageReader's configured buffer format</code>. I switched back to <code>PixelFormat.RGBA_8888</code> and discovered that there is only one plane in the <code>ImageObject</code>. The byte buffer of this plane is of size width*height*4 (one byte for R,G,B,A respectively). So I tried to convert this to <code>NV21</code> format.</p>
  
  <p>I modified code from <a href="https://stackoverflow.com/a/13055615/1360853">this answer</a> to produce the following function:</p>
  
  <pre><code>void RGBtoNV21(byte[] yuv420sp, byte[] argb, int width, int height) {
      final int frameSize = width * height;
  
      int yIndex = 0;
      int uvIndex = frameSize;
  
      int A, R, G, B, Y, U, V;
      int index = 0;
      int rgbIndex = 0;
  
      for (int i = 0; i &lt; height; i++) {
          for (int j = 0; j &lt; width; j++) {
  
              R = argb[rgbIndex++];
              G = argb[rgbIndex++];
              B = argb[rgbIndex++];
              A = argb[rgbIndex++];
  
              // RGB to YUV conversion according to
              // https://en.wikipedia.org/wiki/YUV#Y.E2.80.B2UV444_to_RGB888_conversion
              Y = ((66 * R + 129 * G + 25 * B + 128) &gt;&gt; 8) + 16;
              U = ((-38 * R - 74 * G + 112 * B + 128) &gt;&gt; 8) + 128;
              V = ((112 * R - 94 * G - 18 * B + 128) &gt;&gt; 8) + 128;
  
              // NV21 has a plane of Y and interleaved planes of VU each sampled by a factor
              // of 2 meaning for every 4 Y pixels there are 1 V and 1 U.
              // Note the sampling is every other pixel AND every other scanline.
              yuv420sp[yIndex++] = (byte) ((Y &lt; 0) ? 0 : ((Y &gt; 255) ? 255 : Y));
              if (i % 2 == 0 &amp;&amp; index % 2 == 0) {
                  yuv420sp[uvIndex++] = (byte) ((V &lt; 0) ? 0 : ((V &gt; 255) ? 255 : V));
                  yuv420sp[uvIndex++] = (byte) ((U &lt; 0) ? 0 : ((U &gt; 255) ? 255 : U));
              }
              index++;
          }
      }
  }
  </code></pre>
  
  <p>and invoke it using:</p>
  
  <pre><code>int mWidth = mImage.getWidth();
  int mHeight = mImage.getHeight();
  
  byte[] rgbaBytes = new byte[mWidth * mHeight * 4];
  mImage.getPlanes()[0].getBuffer().get(rgbaBytes);
  mImage.close();
  
  byte[] yuv = new byte[mWidth * mHeight * 3 / 2];
  RGBtoNV21(yuv, rgbaBytes, mWidth, mHeight);
  </code></pre>
  
  <p>Here <code>mImage</code> is an <code>Image</code> object produced by my <code>ImageReader</code>.</p>
  
  <p>Yet this produces a result similar to this image: </p>
  
  <p><img src="https://gyazo.com/e8d3bbc14eb93970b4434a2ddf95c670.png" alt="malformed image"></p>
  
  <p>which is clearly malformed. Seems like my conversion is off but I cannot figure what exactly.</p>