Hugging Face's logo

Spaces:
SEA-AI
/
det-metrics
Running

App Files Community
det-metrics / README.md
franzi2505's picture
franzi2505
Update README.md
bff0759 verified
preview code
|
raw
history blame
9.29 kB
metadata
title: Detection Metrics
tags:
  - evaluate
  - metric
description: >-
  Compute multiple object detection metrics at different bounding box area
  levels.
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false
emoji: 🕵️

Metric Card for Detection Metric

Metric Description

This metric can be used to calculate object detection metrics. It has an option to calculate the metrics at different levels of bounding box sizes, so that more insight is provided into the performance for different objects. It is adapted from the base of pycocotools metrics.

How to Use 

>>> import evaluate
>>> from seametrics.fo_to_payload.utils import fo_to_payload
>>> b = fo_to_payload(
>>>         dataset="SAILING_DATASET_QA",
>>>         gt_field="ground_truth_det",
>>>         models=["yolov5n6_RGB_D2304-v1_9C"],
>>>         sequence_list=["Trip_14_Seq_1"],
>>>         data_type="rgb"
>>>    )
>>> module = evaluate.load("SEA-AI/det-metrics.py")
>>> module.add_batch(b)
>>> res = module.compute()
>>> print(res)
{'all': {'range': [0, 10000000000.0],
  'iouThr': '0.00',
  'maxDets': 100,
  'tp': 89,
  'fp': 13,
  'fn': 15,
  'duplicates': 1,
  'precision': 0.8725490196078431,
  'recall': 0.8557692307692307,
  'f1': 0.8640776699029126,
  'support': 104,
  'fpi': 0,
  'nImgs': 22}}

Metric Settings

When loading module: module = evaluate.load("SEA-AI/det-metrics", **params), multiple parameters can be specified.

  • area_ranges_tuples List[Tuple[str, List[int]]]: Different levels of area ranges at which metrics should be calculated. It is a list that contains tuples, where the first element of each tuple should specify the name of the area range and the second element is list specifying the lower and upper limit of the area range. Defaults to [("all", [0, 1e5.pow(2)])].
  • bbox_format Literal["xyxy", "xywh", "cxcywh"]: Bounding box format of predictions and ground truth. Defaults to "xywh".
  • iou_threshold Optional[float]: at which IOU-treshold the metrics should be calculated. IOU-threshold defines the minimal overlap between a ground truth and predicted bounding box so that it is considered a correct prediction. Defaults to 1e-10.
  • class_agnostic bool. Defaults to True. Non-class-agnostic metrics are currently not supported.

Input Values

Add predictions and ground truths to the metric with the function module.add_batches(payload). The format of payload should be as returned by function fo_to_payload() defined in seametrics library. An example of how a payload might look like is:

test_payload = {
    'dataset': 'SAILING_DATASET_QA',
    'models': ['yolov5n6_RGB_D2304-v1_9C'],
    'gt_field_name': 'ground_truth_det',
    'sequences': {
        # sequence 1, 1 frame with 1 pred and 1 gt
        'Trip_14_Seq_1': {
            'resolution': (720, 1280),
            'yolov5n6_RGB_D2304-v1_9C': [[fo.Detection(
                    label='FAR_AWAY_OBJECT',
                    bounding_box=[0.35107421875, 0.274658203125, 0.0048828125, 0.009765625], # tp nr1
                    confidence=0.153076171875
                )]],
                'ground_truth_det': [[fo.Detection(
                    label='FAR_AWAY_OBJECT',
                    bounding_box=[0.35107421875, 0.274658203125, 0.0048828125, 0.009765625]
                )]]
            },
        # sequence 2, 2 frames with frame 1: 2 pred, 1 gt; frame 2: 1 pred 1 gt
        'Trip_14_Seq_2': {
            'resolution': (720, 1280),
            'yolov5n6_RGB_D2304-v1_9C': [
                [
                    fo.Detection(
                        label='FAR_AWAY_OBJECT',
                        bounding_box=[0.389404296875,0.306640625,0.005126953125,0.0146484375], # tp nr 2
                        confidence=0.153076171875
                        ),
                    fo.Detection(
                        label='FAR_AWAY_OBJECT',
                        bounding_box=[0.50390625, 0.357666015625, 0.0048828125, 0.00976562], # fp nr 1
                        confidence=0.153076171875
                        ),
                    fo.Detection(
                        label='FAR_AWAY_OBJECT',
                        bounding_box=[0.455078125, 0.31494140625, 0.00390625, 0.0087890625], # fp nr 2
                        confidence=0.153076171875
                    )
                ],
                [
                    fo.Detection(
                        label='FAR_AWAY_OBJECT',
                        bounding_box=[0.455078125, 0.31494140625, 0.00390625, 0.0087890625], # tp nr 3
                        confidence=0.153076171875
                        )
                ],
                [
                    fo.Detection(
                        label='FAR_AWAY_OBJECT',
                        bounding_box=[0.455078125, 0.31494140625, 0.00390625, 0.0087890625], # fp nr 3
                        confidence=0.153076171875
                        )
                ]
            ],
            'ground_truth_det': [
                # frame nr 1
                [
                    fo.Detection(
                        label='FAR_AWAY_OBJECT',
                        bounding_box=[0.389404296875,0.306640625,0.005126953125,0.0146484375],
                        )
                ],
                # frame nr 2
                [
                    fo.Detection(
                        label='FAR_AWAY_OBJECT',
                        bounding_box=[0.455078125, 0.31494140625, 0.00390625, 0.0087890625],
                        confidence=0.153076171875
                    ),
                    fo.Detection(
                        label='FAR_AWAY_OBJECT',
                        bounding_box=[0.35107421875, 0.274658203125, 0.0048828125, 0.009765625], # missed nr 1
                        confidence=0.153076171875
                    )
                ],
                # frame nr3
                [
                ], 
            ]
            }
        },
    "sequence_list": ["Trip_14_Seq_1", 'Trip_14_Seq_2']
}

Optionally, you can pass the model as string that should be evaluated, via model=model_str. By default, it will evaluate the first model, i.e. model = payload["models"][0].

Output Values

The metric outputs a dictionary that contains sub-dictionaries for each name of the specified area ranges. Each sub-dictionary holds performance metrics at the specific area range level:

  • range: corresponding area range
  • iouThr: IOU-threshold used in calculating the metric
  • maxDets: maximum number of detections in calculating the metrics
  • tp: number of true positive predictions
  • fp: number of false positive predictions
  • fn: number of false negative predictions
  • duplicates: number of duplicated bounding box predictions
  • precision: ratio between true positive predictions and positive predictions (tp/(tp+fp))
  • recall: ratio between true positive predictions and actual ground truths (tp/(tp+fn))
  • f1: trades-off precision and recall (2*(precision*recall)/(precision+recall))
  • support: number of ground truth bounding boxes that are considered in the metric
  • fpi: number of images with predictions but no ground truths
  • nImgs: number of total images considered in calculating the metric

Examples

We can specify different area range levels, at which we would like to compute the metrics.

>>> import evaluate
>>> from seametrics.fo_to_payload.utils import fo_to_payload
>>> area_ranges_tuples = [
        ("all", [0, 1e5 ** 2]),
        ("small", [0 ** 2, 6 ** 2]),
        ("medium", [6 ** 2, 12 ** 2]),  
        ("large", [12 ** 2, 1e5 ** 2])
    ]
>>> payload = fo_to_payload(
        dataset=dataset, 
        gt_field=gt_field, 
        models=model_list
    )
>>> module = evaluate.load(
      "./detection_metric.py",
      iou_thresholds=0.9,
      area_ranges_tuples=area_ranges_tuples
)
>>> module.add_batch(payload)
>>> result = module.compute()
>>> print(result)
{'all': 
    {'range': [0, 10000000000.0],
    'iouThr': '0.00',
    'maxDets': 100,
    'tp': 0, 
    'fp': 3,
    'fn': 1,
    'duplicates': 0,
    'precision': 0.0,
    'recall': 0.0,
    'f1': 0,
    'support': 1,
    'fpi': 1,
    'nImgs': 2
}, 
'small': {
    'range': [0, 36],
    'iouThr': '0.00',
    'maxDets': 100,
    'tp': 0,
    'fp': 1,
    'fn': 1,
    'duplicates': 0,
    'precision': 0.0,
    'recall': 0.0,
    'f1': 0,
    'support': 1,
    'fpi': 1, 
    'nImgs': 2
}, 
'medium': {
    'range': [36, 144],
    'iouThr': '0.00',
    'maxDets': 100,
    'tp': 0,
    'fp': 2,
    'fn': 0,
    'duplicates': 0,
    'precision': 0.0,
    'recall': 0,
    'f1': 0,
    'support': 0,
    'fpi': 2,
    'nImgs': 2
}, 'large': {
    'range': [144, 10000000000.0],
    'iouThr': '0.00',
    'maxDets': 100,
    'tp': -1,
    'fp': -1,
    'fn': -1,
    'duplicates': -1,
    'precision': -1,
    'recall': -1,
    'f1': -1,
    'support': 0,
    'fpi': 0,
    'nImgs': 2
    }
}

Further References

seametrics library: https://github.com/SEA-AI/seametrics/tree/main

Calculating metrics is based on pycoco tools: https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools

Further info about metrics: https://www.analyticsvidhya.com/blog/2020/09/precision-recall-machine-learning/