title: How to convert cv::Mat to pcl::pointcloud with color
body: <p>I am trying to convert a pointcloud with rgb information from pcl format to cv::Mat and back to pcl. I have found <a href="https://stackoverflow.com/questions/32521043/how-to-convert-cvmat-to-pclpointcloud">convert mat to pointcloud</a> on stackoverflow.</p>
  
  <p><strong>Code Updated</strong></p>
  
  <p>I therefore used that code found on stackoverflow as a starting point. And now have the following:</p>
  
  <pre><code>//input pcl::PointCloud&lt;pcl::PointXYZRGB&gt; point_cloud_ptr
  cv::Mat OpenCVPointCloud(3, point_cloud_ptr.points.size(), CV_64FC1);
  OpenCVPointCloudColor = cv::Mat(480, 640, CV_8UC3);
  for(int y=0;y&lt;OpenCVPointCloudColor.rows;y++)
  {
      for(int x=0;x&lt;OpenCVPointCloudColor.cols;x++)
      {
          OpenCVPointCloud.at&lt;double&gt;(0,y*OpenCVPointCloudColor.cols+x) = point_cloud_ptr.points.at(y*OpenCVPointCloudColor.cols+x).x;
          OpenCVPointCloud.at&lt;double&gt;(1,y*OpenCVPointCloudColor.cols+x) = point_cloud_ptr.points.at(y*OpenCVPointCloudColor.cols+x).y;
          OpenCVPointCloud.at&lt;double&gt;(2,y*OpenCVPointCloudColor.cols+x) = point_cloud_ptr.points.at(y*OpenCVPointCloudColor.cols+x).z;
  
          cv::Vec3b color = cv::Vec3b(point_cloud_ptr.points.at(y*OpenCVPointCloudColor.cols+x).b,point_cloud_ptr.points.at(y*OpenCVPointCloudColor.cols+x).g,point_cloud_ptr.points.at(y*OpenCVPointCloudColor.cols+x).r);
  
          OpenCVPointCloudColor.at&lt;cv::Vec3b&gt;(cv::Point(x,y)) = color;
      }
  }
  
  cv::imshow("view 2", OpenCVPointCloudColor);
  cv::waitKey(30);
  </code></pre>
  
  <p>Displaying the image above ensured me that the colours are extracted correctly (the image is compared by eye with the raw image). I then want to convert it back to a pointcloud:</p>
  
  <pre><code>pcl::PointCloud&lt;pcl::PointXYZRGB&gt;::Ptr point_cloud_ptr(new pcl::PointCloud&lt;pcl::PointXYZRGB&gt;);
  
  for(int y=0;y&lt;OpenCVPointCloudColor.rows;y++)
  {
      for(int x=0;x&lt;OpenCVPointCloudColor.cols;x++)
      {
          pcl::PointXYZRGB point;
          point.x = OpencVPointCloud.at&lt;double&gt;(0,y*OpenCVPointCloudColor.cols+x);
          point.y = OpencVPointCloud.at&lt;double&gt;(1,y*OpenCVPointCloudColor.cols+x);
          point.z = OpencVPointCloud.at&lt;double&gt;(2,y*OpenCVPointCloudColor.cols+x);
  
              cv::Vec3b color = OpenCVPointCloudColor.at&lt;cv::Vec3b&gt;(cv::Point(x,y));
          //Try 1 not working
          //uint8_t r = (color[2]); 
          //uint8_t g = (color[1]);
          //uint8_t b = (color[0]);
  
          //Try 2 not working
          //point.r = color[2];
          //point.g = color[1];
          //point.b = color[0];
  
          //Try 3 not working
          //uint8_t r = (color.val[2]);
          //uint8_t g = (color.val[1]);
          //uint8_t b = (color.val[0]);
  
          //test working WHY DO THE ABOVE CODES NOT WORK :/
          uint8_t r = 255; 
          uint8_t g = 0;
          uint8_t b = 0;
          int32_t rgb = (r &lt;&lt; 16) | (g &lt;&lt; 8) | b;
          point.rgb = *reinterpret_cast&lt;float*&gt;(&amp;rgb);
  
          point_cloud_ptr -&gt; points.push_back(point);
      }
  } 
  </code></pre>
  
  <p>Can anybody explain why the values explicitly specified 255,0,0 colours everything red. But if I choose the output from colour image, the cloud is wrongly coloured? </p>
  
  <p>Stumbled upon <a href="https://stackoverflow.com/a/29303673/2076775">this</a>, I cannot see what difference my code has other than then the cloud type is different?</p>
  
  <p><strong>update</strong></p>
  
  <p>reading <a href="http://www.pcl-users.org/Manual-PCL-creation-color-problem-td4033221.html" rel="nofollow noreferrer">this</a> discussion on PCL ended with me switching to xyzrgba (also mentioned on <a href="https://stackoverflow.com/questions/29270544/how-to-display-a-3d-image-when-we-have-depth-and-rgb-mats-in-opencv-captured-f/29273919">stackoverflow</a>). The code I then tried when converting back is:</p>
  
  <pre><code>point.b = OpenCVPointCloudColor.at&lt;cv::Vec3b&gt;(x,y)[0];
  point.g = OpenCVPointCloudColor.at&lt;cv::Vec3b&gt;(x,y)[1];
  point.r = OpenCVPointCloudColor.at&lt;cv::Vec3b&gt;(x,y)[2];
  point.a = 255;
  </code></pre>
  
  <p>The resulting colour cloud is different from the ones created in XYZRGB, but are still wrong :/ WTF?</p>
  
  <p><strong>Extra</strong></p>
  
  <p>Even if I force a red colour into all points <code>OpenCVPointCloudColor</code> using <code>cvCloudColor.at&lt;cv::Vec3f&gt;(x,y) = cv::Vec3f(0,0,255);</code> then reading from <code>OpenCVPointCloudColor</code> still creates wrong colour information in the pcl cloud.</p>