metadata

title: SignWriting Similarity
sdk: gradio
sdk_version: 3.36.1
app_file: app.py
pinned: false
tags:
  - evaluate
  - metric
description: >-
  The Symbol Distance Metric is a novel evaluation metric specifically designed
  for SignWriting,  a visual writing system for signed languages. Unlike
  traditional string-based metrics (e.g., BLEU, chrF),  this metric directly
  considers the visual and spatial properties of individual symbols used in
  SignWriting,  such as base shape, orientation, rotation, and position. It is
  primarily used to evaluate model outputs in  SignWriting transcription and
  translation tasks, offering a similarity score between a predicted and a
  reference sign.

Metric Card for SignWriting Similarity

Metric Description

The Symbol Distance Metric is a novel evaluation metric specifically designed for SignWriting, a visual writing system for signed languages. Unlike traditional string-based metrics (e.g., BLEU, chrF), this metric directly considers the visual and spatial properties of individual symbols used in SignWriting, such as base shape, orientation, rotation, and position. It is primarily used to evaluate model outputs in SignWriting transcription and translation tasks, offering a similarity score between a predicted and a reference sign.

How to Use

The metric is used by passing two SignWriting signs (as sets of symbols) and computing a similarity score that reflects how closely they match in terms of symbol content and layout.

Inputs

hypothesis (List[Symbol]): The output sign, represented as a list of symbols with visual and spatial properties.
reference (List[Symbol]): The gold/reference sign, in the same format.

Output Values

Returns a dictionary like:

{"score": 0.83}

This metric outputs a score between 0 and 1:

1.0: Perfect similarity (identical signs)
0.0: Complete dissimilarity Higher scores are better. A score above 0.8 is typically considered very good for single sign comparisons.

Limitations and Bias

The metric relies on a manually defined distance function for symbol attributes, which may not fully capture perceptual similarity.
Performance has primarily been validated qualitatively; quantitative alignment with human judgment is ongoing.
It assumes symbol independence and uses a Hungarian matching algorithm, which may miss some higher-order structural patterns in complex signs.
Currently more suitable for evaluating single signs than continuous signing sequences.

Citation

Amit Moryossef, Rotem Zilberman, Ohad Langer (2024). Effective Sign Language Evaluation via SignWriting. arXiv:2410.13668