title: Extract pointclouds WITH colour using the Project Tango; i.e. getting the current camera frame
body: <p>I am trying to produce a point cloud where each point has a colour. I can get just the point cloud or I can get the camera to take a picture, but I need them to be as simultaneous as possible. If I could look up an RGB image with a timestamp or call a function to get the current frame when <code>onXYZijAvailable()</code> is called I would be done. I could just go over the points, find out where it would intersect with the image plane and get the colour of that pixel.</p>
  
  <p>As it is now I have not found any way to get the pixel info of an image or get coloured points. I have seen AR apps where the camera is connected to the <code>CameraView</code> and then things are rendered on top, but the camera stream is never touched by the application. </p>
  
  <p>According to <a href="https://stackoverflow.com/questions/32232641/is-it-possible-to-acquire-raw-imu-and-rgb-d-data-with-google-tango">this</a> post it should be possible to get the data I want and synchronize the point cloud and the image plane by a simple transformation. <a href="https://stackoverflow.com/questions/29236110/timing-issues-with-tango-image-frames">This</a> post is also saying something similar. However, I have no idea how to get the RGB data. I cant find any open source projects or tutorials. </p>
  
  <p>The closest I have gotten is finding out when a frame is ready by using this:    </p>
  
  <pre><code>     public void onFrameAvailable(final int cameraId) {
            if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) {
                 //Get the new rgb frame somehow.
            }
       }
  </code></pre>
  
  <p>I am working with the Java API and I would very much like to not delve into JNI and the NDK if at all possible. How can I get the frame that most closely matches the timestamp of my current point cloud?</p>
  
  <p>Thank you for your help.</p>
  
  <p><strong>Update:</strong></p>
  
  <p>I implemented a CPU version of it and even after optimising it a bit I only managed to get .5 FPS on a small point cloud. This is also due to the fact that the colours have to be converted from the android native NV21 colour space to the GPU native RGBA colour space. I could have optimized it further, but I am not going to get a real time effect with this. The CPU on the android device simply can not perform well enough. If you want to do this on more than a few thousand points, go for the extra hassle of using the GPU or do it in post.</p>