title: Hololens 2 & Unity - How to use the CameraToWorldMatrix to correct the position of a hologram?
body: <p>I'm using the Hololens 2 with Unity 2019.4.6.</p>
  <p>With the PhotoCaptureFrame class, I'm taking a picture, analyzing it to find a specific marker and then I create an hologram where the marker is. My code is working well with a webcam in the Unity Editor. But on the hololens, there is an offset between the hologram and the marker.</p>
  <p>I try to correct it with cameraToWorld matrix (that I get with the photocaptureframe class). If I understand well, it represents the offset between the hololens camera and the RGB camera (the one taking a picture).</p>
  <p>The problem is : I don't know if I'm using well this matrix, my hologram position is always weird... I tried to use this tutorial : <a href="https://forum.unity.com/threads/holographic-photo-blending-with-photocapture.416023/" rel="nofollow noreferrer">https://forum.unity.com/threads/holographic-photo-blending-with-photocapture.416023/</a>
  And the result is wrong too ! The instantiated quad is rotated and behind the original starting point (when I start the app). It doesn't follow the camera.</p>
  <p>Do I need to convert the matrix ?
  Can someone explains to me what I'm doing wrong please ?
  Thanks in advance !</p>
  <p>Simple example to get matrices : It prints out true for the CameraToWorld matrix and The Projection matrix but the matrices are not updated when you move and take a picture from another location.</p>
  <pre><code>using System;
  using System.Collections.Generic;
  using UnityEngine;
  using UnityEngine.Windows.WebCam;
  
  public class HoloLensSnapshotTest : MonoBehaviour
  {
      PhotoCapture m_PhotoCaptureObj;
      CameraParameters m_CameraParameters;
      bool m_CapturingPhoto = false;
  
      void Start()
      {
          Initialize();
      }
  
  
      [ContextMenu(&quot;TakePicture&quot;)]
      public void TakePicture() // called by button for now
      {
          if (m_CapturingPhoto)
          {
              return;
          }
  
          m_CapturingPhoto = true;
          Debug.Log(&quot;Taking picture...&quot;);
          m_PhotoCaptureObj.TakePhotoAsync(OnPhotoCaptured);
      }
  
      void Initialize()
      {
          Debug.Log(&quot;Initializing...&quot;);
          List&lt;Resolution&gt; resolutions = new List&lt;Resolution&gt;(PhotoCapture.SupportedResolutions);
          Resolution selectedResolution = resolutions[0];
  
          m_CameraParameters = new CameraParameters(WebCamMode.PhotoMode);
          m_CameraParameters.cameraResolutionWidth = selectedResolution.width;
          m_CameraParameters.cameraResolutionHeight = selectedResolution.height;
          m_CameraParameters.hologramOpacity = 0.0f;
          m_CameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
  
          PhotoCapture.CreateAsync(false, OnCreatedPhotoCaptureObject);
      }
  
      void OnCreatedPhotoCaptureObject(PhotoCapture captureObject)
      {
          m_PhotoCaptureObj = captureObject;
          m_PhotoCaptureObj.StartPhotoModeAsync(m_CameraParameters, OnStartPhotoMode);
      }
  
      void OnStartPhotoMode(PhotoCapture.PhotoCaptureResult result)
      {
          m_CapturingPhoto = false;
  
          Debug.Log(&quot;Ready&quot;);
      }
  
  
      void OnPhotoCaptured(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
      {
          Matrix4x4 cameraToWorldMatrix;
          bool cameraToWorldMatrixResult = photoCaptureFrame.TryGetCameraToWorldMatrix(out cameraToWorldMatrix);
  
          Matrix4x4 projectionMatrix;
          bool projectionMatrixResult = photoCaptureFrame.TryGetProjectionMatrix(out projectionMatrix);
  
          Debug.Log(&quot;CamToWorld : &quot; + Environment.NewLine + cameraToWorldMatrixResult + Environment.NewLine + $&quot;{cameraToWorldMatrix.ToString()}&quot;);
          Debug.Log(&quot;Projection : &quot; + Environment.NewLine + projectionMatrixResult + Environment.NewLine + $&quot;{projectionMatrix.ToString()}&quot;);
          Debug.Log(&quot;Took picture!&quot;);
  
          m_CapturingPhoto = false;
      }
  }
  </code></pre>
  <p>My matrix are :
  Camera To World :</p>
  <pre><code>0.00650   -0.99959  -0.02805    0.00211
  -0.99965  -0.00577  -0.02588   -0.04999
  0.02571    0.02821  -0.99927   -0.01216
  0.00000    0.00000   0.00000    1.00000
  </code></pre>
  <p>Projection :</p>
  <pre><code>-0.06754   1.52561   0.00000  0.00000
  -2.71531  -0.12021   0.00000  0.00000
  0.05028    0.01975  -1.00401 -0.20040
  0.00000    0.00000   0.00000  1.00000
  </code></pre>