Approach to classify a photo and extract text from it
There are three functional aspects to this project, each of which could be a sub-project, even though they are interrelated. The decoupling of these aspects in the associated R&amp;D will likely improve the rate of development.


Classify images &mdash; in this case, the vendor brand
Locate and orient the text
OCR the text &mdash; in this case, the total


These are certainly feasible in the particular case of invoices from vendors, so the entire project certainly is.

The goal is to produce two fields of information from input images.


Vendor code
Total in some specific monetary unit


That past invoices, the names of the vendors, and the totals of each invoice are available as training data may be helpful. Whether the volume of training data is sufficient can only be determined through a very complicated application of theory to some of the data metrics or by trial. It is recommended to do both.


  What algorithms would you guys recommend?


It would be irresponsible to define an AI design in terms of what people recommend for algorithm. The algorithms that guys recommend are already covered by the path names for files in the algorithms directory paths, the example's directory paths, and the sections of the documentation of any good AI framework. Let's first talk about the design and then the algorithm options, so there is more of a basis for algorithm selection than the largely random reading selections of the members of a site.

The models involved and which can be parameterized such that they can be tuned through training are partly defined by the three sub-projects above.

Analysis

For this project, there is no reason for the AI to recognize which text on the invoice is the vendor name. If there are a hundred vendor invoice types, the speed in which a single person can identify the key boundaries of each invoice type is orders of magnitude faster than the work necessary to develop a completely general algorithmic approach for automating that work. It would only be resource thrifty to develop such automation if there were thousands of vendors and constantly varying templates for invoices, which is likely in the case of personal finance.

These are the key bounds of the form quadrilaterals (four sided polygons) within which the following objects lie.


Vendor name text box
Total amount text box
Rectangle around the entire form


Digitizing the twelve points through one of the labelling programs is a model much more likely to produce a reliable system than the representation of three rectangular bounding boxes and tilts. This is because tilt angle requires at least two adjacent points anyway and the aspect ratio cannot be held constant in real scanning scenarios when the scanner might be replaced or produce different ratios with wear or the smoothness or wear characteristics of the paper invoice.

Doing all three things using a network would require more than one person's invoices for a decade, unless the person is a billionaire compulsive buyer with a team of secretaries doing scanning and data entry.

The models are then these.


Digitization of a quadrilateral document with varying contrast, brightness, tilt, horizontal pixels per inch, vertical pixels per inch, relative location and size of vendor name, and relative location and size of invoice total
Numeric characters and other monetary characters in a rectangular box
Brand image for the vendor, which may include a logo and type of arbitrary and possibly unique font


Now we can talk about artificial network approaches.


Adjusting for contrast, brightness, tilt, resolution, locations, and sizes will require a customized input, loss function, and layer arrangement based on the geometry involved. A GRU network may gain some advantages if the documents can be ordered chronologically, since the collection then becomes a time series, the trends of which can be exploited.
Monetary values can best be done using OCR libraries.
Recognizing the brand is probably best done with CNN as a categorization machine.


The system must indicate if the vendor and total are not found, in which case a new template is indicated, and the twelve points must be digitized for this new type.

It is possible to do this with one single deep convolutional network, but, again, the data set would need to be augmented. The one other way to do this is to create a filled-out-invoice generator to produce the volume of stochastically variable data across the various variability dimensions listed above to train the deep CNN.