BERT_rationale_benchmark/metrics.py
def partial_match_score(truth: List[Rationale], pred: List[Rationale], thresholds: List[float]) -> List[Dict[str, Any]]:
    """Computes a partial match F1

    Computes an instance-level (annotation) micro- and macro-averaged F1 score.
    True Positives are computed by using intersection-over-union and
    thresholding the resulting intersection-over-union fraction.

    Micro-average results are computed by ignoring instance level distinctions
    in the TP calculation (and recall, and precision, and finally the F1 of
    those numbers). Macro-average results are computed first by measuring
    instance (annotation + document) precisions and recalls, averaging those,
    and finally computing an F1 of the resulting average.
    """

    ann_to_rat = _keyed_rationale_from_list(truth)
    pred_to_rat = _keyed_rationale_from_list(pred)

    num_classifications = {k:len(v) for k,v in pred_to_rat.items()}
    num_truth = {k:len(v) for k,v in ann_to_rat.items()}
    ious = defaultdict(dict)
    for k in set(ann_to_rat.keys()) | set(pred_to_rat.keys()):
        for p in pred_to_rat.get(k, []):
            best_iou = 0.0
            for t in ann_to_rat.get(k, []):
                num = len(set(range(p.start_token, p.end_token)) & set(range(t.start_token, t.end_token)))
                denom = len(set(range(p.start_token, p.end_token)) | set(range(t.start_token, t.end_token)))
                iou = 0 if denom == 0 else num / denom
                if iou > best_iou:
                    best_iou = iou
            ious[k][p] = best_iou
    scores = []
    for threshold in thresholds:
        threshold_tps = dict()
        for k, vs in ious.items():
            threshold_tps[k] = sum(int(x >= threshold) for x in vs.values())
        micro_r = sum(threshold_tps.values()) / sum(num_truth.values()) if sum(num_truth.values()) > 0 else 0
        micro_p = sum(threshold_tps.values()) / sum(num_classifications.values()) if sum(num_classifications.values()) > 0 else 0
        micro_f1 = _f1(micro_r, micro_p)
        macro_rs = list(threshold_tps.get(k, 0.0) / n if n > 0 else 0 for k, n in num_truth.items())
        macro_ps = list(threshold_tps.get(k, 0.0) / n if n > 0 else 0 for k, n in num_classifications.items())
        macro_r = sum(macro_rs) / len(macro_rs) if len(macro_rs) > 0 else 0
        macro_p = sum(macro_ps) / len(macro_ps) if len(macro_ps) > 0 else 0
        macro_f1 = _f1(macro_r, macro_p)
        scores.append({'threshold': threshold,
                       'micro': {
                            'p': micro_p,
                            'r': micro_r,
                            'f1': micro_f1
                       },
                       'macro': {
                            'p': macro_p,
                            'r': macro_r,
                            'f1': macro_f1
                       },
                       })
    return scores
Computes a partial match F1

    Computes an instance-level (annotation) micro- and macro-averaged F1 score.
    True Positives are computed by using intersection-over-union and
    thresholding the resulting intersection-over-union fraction.

    Micro-average results are computed by ignoring instance level distinctions
    in the TP calculation (and recall, and precision, and finally the F1 of
    those numbers). Macro-average results are computed first by measuring
    instance (annotation + document) precisions and recalls, averaging those,
    and finally computing an F1 of the resulting average.
    
Computes a partial match F1
Computes an instance-level (annotation) micro- and macro-averaged F1 score.
True Positives are computed by using intersection-over-union and
thresholding the resulting intersection-over-union fraction.

Micro-average results are computed by ignoring instance level distinctions
in the TP calculation (and recall, and precision, and finally the F1 of
those numbers). Macro-average results are computed first by measuring
instance (annotation + document) precisions and recalls, averaging those,
and finally computing an F1 of the resulting average.
Computes a partial match F1 Computes an instance - level ( annotation ) micro - and macro - averaged F1 score . True Positives are computed by using intersection - over - union and thresholding the resulting intersection - over - union fraction . Micro - average results are computed by ignoring instance level distinctions in the TP calculation ( and recall and precision and finally the F1 of those numbers ) . Macro - average results are computed first by measuring instance ( annotation + document ) precisions and recalls averaging those and finally computing an F1 of the resulting average .
def partial_match_score(truth: List[Rationale], pred: List[Rationale], thresholds: List[float]) -> List[Dict[str, Any]]:
    ann_to_rat = _keyed_rationale_from_list(truth)
    pred_to_rat = _keyed_rationale_from_list(pred)
    num_classifications = {k:len(v) for k,v in pred_to_rat.items()}
    num_truth = {k:len(v) for k,v in ann_to_rat.items()}
    ious = defaultdict(dict)
    for k in set(ann_to_rat.keys()) | set(pred_to_rat.keys()):
        for p in pred_to_rat.get(k, []):
            best_iou = 0.0
            for t in ann_to_rat.get(k, []):
                num = len(set(range(p.start_token, p.end_token)) & set(range(t.start_token, t.end_token)))
                denom = len(set(range(p.start_token, p.end_token)) | set(range(t.start_token, t.end_token)))
                iou = 0 if denom == 0 else num / denom
                if iou > best_iou:
                    best_iou = iou
            ious[k][p] = best_iou
    scores = []
    for threshold in thresholds:
        threshold_tps = dict()
        for k, vs in ious.items():
            threshold_tps[k] = sum(int(x >= threshold) for x in vs.values())
        micro_r = sum(threshold_tps.values()) / sum(num_truth.values()) if sum(num_truth.values()) > 0 else 0
        micro_p = sum(threshold_tps.values()) / sum(num_classifications.values()) if sum(num_classifications.values()) > 0 else 0
        micro_f1 = _f1(micro_r, micro_p)
        macro_rs = list(threshold_tps.get(k, 0.0) / n if n > 0 else 0 for k, n in num_truth.items())
        macro_ps = list(threshold_tps.get(k, 0.0) / n if n > 0 else 0 for k, n in num_classifications.items())
        macro_r = sum(macro_rs) / len(macro_rs) if len(macro_rs) > 0 else 0
        macro_p = sum(macro_ps) / len(macro_ps) if len(macro_ps) > 0 else 0
        macro_f1 = _f1(macro_r, macro_p)
        scores.append({'threshold': threshold,
                       'micro': {
                            'p': micro_p,
                            'r': micro_r,
                            'f1': micro_f1
                       },
                       'macro': {
                            'p': macro_p,
                            'r': macro_r,
                            'f1': macro_f1
                       },
                       })
    return scores
def partial_match_score ( truth : List [ Rationale ] , pred : List [ Rationale ] , thresholds : List [ float ] ) -> List [ Dict [ str , Any ] ] : ann_to_rat = _keyed_rationale_from_list ( truth ) pred_to_rat = _keyed_rationale_from_list ( pred ) num_classifications = { k : len ( v ) for k , v in pred_to_rat . items ( ) } num_truth = { k : len ( v ) for k , v in ann_to_rat . items ( ) } ious = defaultdict ( dict ) for k in set ( ann_to_rat . keys ( ) ) | set ( pred_to_rat . keys ( ) ) : for p in pred_to_rat . get ( k , [ ] ) : best_iou = 0.0 for t in ann_to_rat . get ( k , [ ] ) : num = len ( set ( range ( p . start_token , p . end_token ) ) & set ( range ( t . start_token , t . end_token ) ) ) denom = len ( set ( range ( p . start_token , p . end_token ) ) | set ( range ( t . start_token , t . end_token ) ) ) iou = 0 if denom == 0 else num / denom if iou > best_iou : best_iou = iou ious [ k ] [ p ] = best_iou scores = [ ] for threshold in thresholds : threshold_tps = dict ( ) for k , vs in ious . items ( ) : threshold_tps [ k ] = sum ( int ( x >= threshold ) for x in vs . values ( ) ) micro_r = sum ( threshold_tps . values ( ) ) / sum ( num_truth . values ( ) ) if sum ( num_truth . values ( ) ) > 0 else 0 micro_p = sum ( threshold_tps . values ( ) ) / sum ( num_classifications . values ( ) ) if sum ( num_classifications . values ( ) ) > 0 else 0 micro_f1 = _f1 ( micro_r , micro_p ) macro_rs = list ( threshold_tps . get ( k , 0.0 ) / n if n > 0 else 0 for k , n in num_truth . items ( ) ) macro_ps = list ( threshold_tps . get ( k , 0.0 ) / n if n > 0 else 0 for k , n in num_classifications . items ( ) ) macro_r = sum ( macro_rs ) / len ( macro_rs ) if len ( macro_rs ) > 0 else 0 macro_p = sum ( macro_ps ) / len ( macro_ps ) if len ( macro_ps ) > 0 else 0 macro_f1 = _f1 ( macro_r , macro_p ) scores . append ( { 'threshold' : threshold , 'micro' : { 'p' : micro_p , 'r' : micro_r , 'f1' : micro_f1 } , 'macro' : { 'p' : macro_p , 'r' : macro_r , 'f1' : macro_f1 } , } ) return scores
Computes a partial match F1
Computes an instance-level (annotation) micro- and macro-averaged F1 score.
Computes a partial match F1 Computes an instance - level ( annotation ) micro - and macro - averaged F1 score .
"""Computes a partial match F1

    Computes an instance-level (annotation) micro- and macro-averaged F1 score.
    True Positives are computed by using intersection-over-union and
    thresholding the resulting intersection-over-union fraction.

    Micro-average results are computed by ignoring instance level distinctions
    in the TP calculation (and recall, and precision, and finally the F1 of
    those numbers). Macro-average results are computed first by measuring
    instance (annotation + document) precisions and recalls, averaging those,
    and finally computing an F1 of the resulting average.
    """
truth
List[Rationale]
pred
List[Rationale]
thresholds
List[float]