title: Learning method for detecting relevant fields in forms (image format)
body: <p>So I have been working on an application where users can upload a (scanned) PDF file representing some form, draw bounding boxes around the fields they are interested in and have those contents OCR'd and returned in structured text format. Because drawing bounding boxes is kind of a drag, I was thinking about a way to decrease the work required by the user; i.e. already offer him/her an auto-detected division of fields in de form. I started researching the matter and found some interesting approaches, mostly based on computer vision algorithms. However, as this application might be used frequently in the future and thus a lot of bounding boxes will be drawn by users, it'd almost seem like a waste to me to not try and use this dataset to apply a learning method to. So I started looking over a lot of different forms and noticed that most of them are structured by borders in a way like this:</p>
  
  <p><a href="https://i.stack.imgur.com/zPGLW.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/zPGLW.jpg" alt="enter image description here"></a></p>
  
  <p>A few observations here: boxes that are filled 100% with text are usually not requested for extraction as they represent terms/conditions/disclaimers/etc. Boxes that are (mostly) empty are also not requested as they mostly indicate unrelevant fields. The only interesting boxes appear to be those with a label in the top/left and some content in the body of the box.</p>
  
  <p>It should of course also be said that not every form is as nicely structured with borders as the above one. Some use only a single dividing border (i.e. either horizontally or vertically) between fields and sometimes there are no borders at all.</p>
  
  <p>Since we are working with images, I started looking into object recognition and tried out YOLOv2 (convolutional neural network) that I let train for a night on a dataset of 100 forms (I realize this dataset is still too small and since I trained on my CPU, that I haven't trained it long enough either). Anyway, I was hoping that the fact that all training fields had a border and some content would quickly help the system in finding bordered boxes itself. The results were however quite disappointing (avg loss/error = 9.6) so far. I started thinking about this and then realised that if users skip drawing certain fields that were perfectly fine bordered boxes, it would confuse the neural network in its learning process, am I right about this?</p>
  
  <p>As for the remainder of my question: do you guys think object recognition is the way to go here or is it way too confusing for the system given the nature of such forms? If so, would it still be the case if I applied for example some filter(s) to try and "blur" text together, making the boxes look a lot more like each other? Or else, given this dataset of coordinates of (most) relevant boxes per document, what would instead be a better learning method to apply? Perhaps even a method that would not be based so much on the presence of a border?</p>
  
  <p>Keep in mind that the only requirement I ask for is to be able to use the user drawn bounding boxes as dataset to continuously improve the system.</p>
  
  <p>Thank you all for your time!</p>