title: How to implement 4-point perspective transform using HTML5 canvas & three.js?
body: <p>First, a visual example of what I am trying to achieve:</p>
  
  <p><a href="https://i.stack.imgur.com/5Uo1h.png" rel="noreferrer"><img src="https://i.stack.imgur.com/5Uo1h.png" alt="example"></a></p>
  
  <p>(Photo credit: <a href="https://unsplash.com/photos/pGcqw1ARGyg" rel="noreferrer">https://unsplash.com/photos/pGcqw1ARGyg</a>)</p>
  
  <h2>The short (tl;dr) question</h2>
  
  <p>Using HTML5 video &amp; canvas, how can I perform a 4-point perspective transform so that I can render just the "TV screen" part of the frame in the canvas? Why doesn't <a href="https://bitbucket.org/mattwilson1024/perspective-transform/src/wrong-transform/" rel="noreferrer">my implementation</a> show the correct area?</p>
  
  <h2>Background about what I'm trying to achieve</h2>
  
  <p>I am trying to build a web page which works as follows:</p>
  
  <ol>
  <li>The user points their webcam towards a TV, so that it is somewhere in the frame (but potentially at any angle)</li>
  <li>Using HTML5 video &amp; canvas, the webcam is captured and previewed on the web page</li>
  <li>The user is able to define (by clicking on the preview) where the 4 corners of the TV screen are (4 pairs of x/y coordinates)</li>
  <li>** The video is warped (using some kind of perspective transform) so that the canvas <em>only</em> shows the part of the image for their actual TV screen (not the whole webcam view) **</li>
  <li>Some processing is then performed on the image (for example, identifying most prominent colours). This part outside the scope of this question, other than pointing out that I will want to be able to access the content/pixels of an HTML5 canvas at the end.</li>
  </ol>
  
  <p>The part I am struggling with is step 4. In order to make sure that I am only processing the relevant part of the image for each frame of the video, it is important that I "warp" the image so that it only shows the "TV screen" area and not the whole webcam picture.</p>
  
  <p>Having done a bit of reading up, my understanding is that:</p>
  
  <ul>
  <li>This requires some kind perspective transform and, because the webcam can be at any angle and we are not dealing with parallel lines, a 3-dimensional transform is required and 2D will not suffice. This is because a 2D transform (translate/rotate/scale/skew) would not be able to deal with the converging sides.</li>
  <li>HTML5 canvas is a two dimensional context and can therefore only support 2D transforms, not a 3D transform. Because I need a solution that works with <code>canvas</code>, I can't simply use a 3D CSS transform (e.g. <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/transform-function/matrix3d" rel="noreferrer">https://developer.mozilla.org/en-US/docs/Web/CSS/transform-function/matrix3d</a>). This suggests that maybe WebGL is more what I need to deal with the 3D aspect.</li>
  </ul>
  
  <h2>What I've attempted so far</h2>
  
  <p>With that in mind, I attempted the following approach:</p>
  
  <p>a) Capture the webcam using a <code>video</code> tag</p>
  
  <p>b) Using <a href="https://threejs.org/" rel="noreferrer">three.js</a>, create a 3D scene which is rendered into a <code>canvas</code> element (so that I can perform my image processing on the resultant canvas contents)</p>
  
  <p>c) The three.js scene consists of:
    - a flat mesh containing which shows the video on one side using a <code>VideoTexture</code>. 
    - a perspective camera, initially positioned so that it shows the whole webcam image</p>
  
  <p>d) Allow the user to click the four corner points to define where their TV is, work out what the x/y coordinates are and save them</p>
  
  <p>e) Calculate a perspective transform which would "stretch" the image out so that the correct area "fills the frame". In other words, stretch the four clicked "TV corner" points to the four corners of the viewport. I have been using this library: <a href="https://github.com/jlouthan/perspective-transform" rel="noreferrer">https://github.com/jlouthan/perspective-transform</a> to calculate this.</p>
  
  <p>f) My thinking is that, if the appropriate transform is applied to the mesh containing the video, and the camera stays in a fixed position, then the output canvas would contain the required image when looking at it in 2D.</p>
  
  <h2>Link to my current (broken) implementation</h2>
  
  <p>Here is a link to my current attempt at the above. It shows the video and allows you to click the four corners. It seems like it works if you click points around the origin (in the centre) but the problem is that it shows the wrong area if you choose areas elsewhere in the image.</p>
  
  <p><a href="https://bitbucket.org/mattwilson1024/perspective-transform/src/wrong-transform/" rel="noreferrer">https://bitbucket.org/mattwilson1024/perspective-transform/src/master/</a></p>
  
  <h2>Summing up</h2>
  
  <p><strong>I'd be really grateful for any help working out why this isn't working as I expected, or for any pointers on whether there is a better/easier approach to achieve what I need.</strong></p>