README.md

---
title: SignWriting Similarity
sdk: gradio
sdk_version: 3.36.1
app_file: app.py
pinned: false
tags:
- evaluate
- metric
description: >-
  The Symbol Distance Metric is a novel evaluation metric specifically designed for SignWriting, 
  a visual writing system for signed languages. Unlike traditional string-based metrics (e.g., BLEU, chrF), 
  this metric directly considers the visual and spatial properties of individual symbols used in SignWriting, 
  such as base shape, orientation, rotation, and position. It is primarily used to evaluate model outputs in 
  SignWriting transcription and translation tasks, offering a similarity score between a predicted and a reference sign.
---

# Metric Card for SignWriting Similarity

## Metric Description
The Symbol Distance Metric is a novel evaluation metric specifically designed for SignWriting, a visual writing system for signed languages. Unlike traditional string-based metrics (e.g., BLEU, chrF), this metric directly considers the visual and spatial properties of individual symbols used in SignWriting, such as base shape, orientation, rotation, and position. It is primarily used to evaluate model outputs in SignWriting transcription and translation tasks, offering a similarity score between a predicted and a reference sign.

## How to Use
The metric is used by passing two SignWriting signs (as sets of symbols) and computing a similarity score that reflects how closely they match in terms of symbol content and layout.

### Inputs

* **hypothesis** *(List\[Symbol]):* The output sign, represented as a list of symbols with visual and spatial properties.
* **reference** *(List\[Symbol]):* The gold/reference sign, in the same format.

### Output Values

Returns a dictionary like:

```python
{"score": 0.83}
```

This metric outputs a score between 0 and 1:

* **1.0**: Perfect similarity (identical signs)
* **0.0**: Complete dissimilarity
  Higher scores are better. A score above 0.8 is typically considered very good for single sign comparisons.

## Limitations and Bias

* The metric relies on a manually defined distance function for symbol attributes, which may not fully capture perceptual similarity.
* Performance has primarily been validated qualitatively; quantitative alignment with human judgment is ongoing.
* It assumes symbol independence and uses a Hungarian matching algorithm, which may miss some higher-order structural patterns in complex signs.
* Currently more suitable for evaluating single signs than continuous signing sequences.

## Citation

Amit Moryossef, Rotem Zilberman, Ohad Langer (2024). *Effective Sign Language Evaluation via SignWriting*. [arXiv:2410.13668](https://arxiv.org/abs/2410.13668)