Document Understanding Subnet: Validation

We are excited to outline the validation process that will be pivotal in the operation of our Document Understanding Subnet on the #Bittensor network. This process is designed to assess and ensure the quality and efficiency of the mining operations carried out by the network’s miners.

VALIDATION COMPONENTS AND PROCEDURE: Validators play a critical role in maintaining the integrity and accuracy of the subnet's operations. Here’s how the validation process works:

1. Database of Annotated Images: A comprehensive database will store all annotated images along with their corresponding ground truths.
2. Document Assignment and Evaluation: Validators will select a random document image from the database and assign it to a miner for processing. They will then evaluate the miner’s predictions focusing on both accuracy and the time taken to process the document.

TIME DECISION CRITERIA: To ensure fairness and objectivity in performance evaluation, specific metrics are set:

1. Standard Time Calculation (Tn): We will conduct experiments to determine a standard processing time, Tn, on typical machines.
2. Time Score Calculation: Time taken by the miner (Tt) will be compared against Tn using the formula:
• Time Score = Tn / (Tn + Tt + ϵ)
• The value of ϵ (a small constant) can be adjusted to fine-tune the scoring.

ACCURACY CALCULATION: Accuracy is assessed based on the precision of box detection and text extraction:

1. For Individual Box-Text Pairs:
• Checked Box Score (CBS): If the overlap between the predicted and actual checked box is greater than 0.95, a score of 1 is assigned. The score decreases gradually if the overlap is between 0.7 and 0.95, and any overlap less than 0.7 receives a score of 0.
• Text Similarity Score (TS): Similarity between detected text and actual text—if 100% match, score is 1; otherwise, it decreases gradually.
• Individual Pair Score = (CBS + TS) / 2

2. For All Box-Text Pairs:
• Accuracy Score = (score-1 + score-2 + ... + score-n) / n

FINAL SCORE DETERMINATION: The final score awarded to a miner is a weighted average of the Time Score and Accuracy Score:

• Final Score = 0.3Time Score + 0.7Accuracy Score
• The weights of these scores can be adjusted based on their relative importance.

Validators ensure that the subnet not only runs efficiently but also maintains high standards of data accuracy and processing speed. This rigorous validation mechanism allows the $TATSU Document Understanding Subnet to operate reliably and effectively within the Bittensor network.
#Bittensor