Evaluating information extraction from structured documents I'm trying to find metrics to evaluate multiple algorithms for key information extraction from already OCRed invoices. For instance, such an algorithm, given an invoice, could find that:

{
    "company": "STARBUCKS STORE #10208",
    "date": "14/03/2015",
    "address": "11302 EUCLID AVENUE, CLEVELAND, OH (216) 229-0749",
    "total": "4.95",
}


Formally, I defined the task as a function f that for a given invoice document D returns a dictionary of entries: {field1:value1,field2:value2,...}.

Initially, I thought about using precision, recall and F1 measures, assuming that each dictionary entry is marked as correct if both a field and its value matches exactly the ground truth. I borrowed this approach from ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction. 

However, the problem with metrics based on exact string match is that they penalize even small errors: missing/redundant words, single differences in characters and spacing. Those errors may stem not only from the imperfect information extraction algorithm but also from faulty OCR algorithm. Unfortunately, I am not able to test information extraction in isolation (i.e., given 100% perfectly OCRed input).

Therefore, I am looking for some more relaxed metrics. What are my possibilities? What has been used in this field?

Some ideas that come to my mind:


calculate distance between two texts using Levenstein distance on character/word
consider two texts matching if distance is smaller than threshold X.
or include information about the distance in the final measures, e.g. normalize it and micro-average over each document.
I have worked with Structured text using OCR. OCR is prone to errors even while reading the content and a slight change in string arrangement would lead to false positives. I used Cosine Similarity and LEvenstein distance. I curated the data with pip install ftfy, Flashtext fuzzywuzzy ,chardet and set a threshold of 97% giving with 2% buffer for errors which might be unintentional/Typos or Inability of OCR to read the character etc. With this approach you are just relaxing on the 2% error and you can still use the same Precision, Recall or Specificity and Accuracy metrics. And we based on new data, you can gather observations and adjust your threshold on run time. And yes, Make your threshold configurable than hard coded which will be easier to run multiple experiments.

I hope this helps.