title: ideas on quadrangle/rectangle detection using convolutional neural networks
body: <p>I'v been trying to do quadrangle detection and localization for weeks, my goal is to have a robust way of getting the 4 points of an quadrangle(rectangle), so I can apply projective transform to an Image then attach it to the source image. I have try the classic opencv contour method, and also using hough transform to find lines then calculate intersections, those two methods is unusable when apply it to real life images.
  <a href="https://i.stack.imgur.com/tJAlG.png" rel="noreferrer"><img src="https://i.stack.imgur.com/tJAlG.png" alt="quadrangle detection"></a></p>
  
  <p>So I turn to CNN for help, but currently i haven't find any one try to use CNN to solve this simple problem. </p>
  
  <p>My first attempt is to use state-of-art object detection and localization methods to get quadrangle's bounding box so i can narrow the search of 4 points, then use image processing &amp; computer vision methods to further the search for 4 points. but after trying YOLOv2 and Faster-RCNN, the prediction accuracy is not ideal.</p>
  
  <p>So I'm wondering if there is any idea i can do this end to end, training and feedforward all using a single neural network. it also must be able to deal with occlusion reasonably well.</p>
  
  <p>Currently my idea is to remove the fc-layers and make a huge activation map that has the same width and height as the first input layer(eg. 448x448) then optimize the 4 most highly activated areas, using argmax to get the position. but this method only works for one quadrangle it doesn't work well with corner occlusions as well.</p>
  
  <p>I'll be appreciated if anyone can provide any suggestions. Thanks a lot!</p>