Added examples and input/outputs in README

README.md

@@ -15,37 +15,89 @@ pinned: false
 
 # Metric Card for Phone Distance
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
-
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+Measures of distance in terms of articulatory phonological features can help understand differences between strings in the International Phonetic Alphabet (IPA) in a linguistically motivated way. 
+This is useful when evaluating speech recognition or orthographic to IPA conversion tasks. These are Levenshtein distances for comparing strings where the smallest unit of measurement is based on phones or articulatory phonological features, rather than Unicode characters. 
 
 ## How to Use
-*Give general statement of how to use the metric*
 
-*Provide simplest possible example for using the metric*
+```python
+import evaluate
+phone_distance = evaluate.load("ginic/phone_distance")
+phone_distance.compute(predictions=["bob", "ði"], references=["pop", "ðə"])
+```
 
 ### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+- **predictions** (`list` of `str`): Transcriptions to score.
+- **references** (`list` of `str`) : Reference strings serving as ground truth.
+- **feature_model** (`str`): Set which panphon.distance.Distance feature parsing model is used, choose from `"strict"`, `"permissive"`, `"segment"`. Defaults to `"segment"`. 
+- **is_normalize_pfer** (`bool`): Set to `True `to normalize PFER by the largest number of phones in the prediction, reference pair. Defaults to `False`. When this is used PFER will no longer obey the triangle inequality.
 
-### Output Values
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+### Output Values
+The computation returns a dictionary with the following key and values:
+ - **phone_error_rates** (`list` of `float`): Phone error rate (PER) gives edit distance in terms of phones for each prediction-reference pair, rather than Unicode characters, since phones can consist of multiple characters. It is normalized by the number of phones of the reference string. The result with have the same length as the input prediction and reference lists.
+ - **mean_phone_error_rate** (`float`): Overall mean of PER.
+ - **phone_feature_error_rates** (`list` of `float`): Phone feature error rate (PFER) is Levenshtein distance between strings where distance between individual phones is computed using Hamming distance between phonetic features for each prediction-reference pair. By default it is a metric that obeys the triangle equality, but can also be normalized by number of phones. 
+ - **mean_phone_feature_error_rates** (`float`):  Overall mean of PFER.
+ - **feature_error_rates** (`list` of `float`): Feature error rate (FER) is the edit distance in terms of articulatory features normalized by the number of phones in the reference, computed for each prediction-reference pair. 
+ - **mean_feature_error_rates** (`float`): Overall mean of FER.
 
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
 
 #### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
+[Universal Automatic Phonetic Transcription into the International Phonetic Alphabet (Taguchi et al.)](https://www.isca-archive.org/interspeech_2023/taguchi23_interspeech.html) reported an overall PER of 0.21 and PFER of 0.057 on supervised phonetic transcription of in-domain languages, a PER of 0.632 and PFER of 0.213 on zero-shot phonetic transcription of languages not seen in training data. On the zero-shot languages they also reported inter-annotator scores between human annotators as PER 0.533 and PFER 0.196.
 
 ### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+
+Simplest use case to compute phone error rates between two IPA strings: 
+```python
+>>> phone_distance.compute(predictions=["bob", "ði", "spin"], references=["pop", "ðə", "spʰin"])
+{'phone_error_rates': [0.6666666666666666, 0.5, 0.25], 'mean_phone_error_rate': 0.47222222222222215, 'phone_feature_error_rates': [0.08333333333333333, 0.125, 0.041666666666666664], 'mean_phone_feature_error_rates': 0.08333333333333333, 'feature_error_rates': [0.027777777777777776, 0.0625, 0.30208333333333337], 'mean_feature_error_rates': 0.13078703703703706}
+```
+
+Normalize phone feature error rate by the length of the reference string: 
+```python
+>>> phone_distance.compute(predictions=["bob", "ði"], references=["pop", "ðə"], is_normalize_pfer=True)
+{'phone_error_rates': [0.6666666666666666, 0.5], 'mean_phone_error_rate': 0.5833333333333333, 'phone_feature_error_rates': [0.027777777777777776, 0.0625], 'mean_phone_feature_error_rates': 0.04513888888888889, 'feature_error_rates': [0.027777777777777776, 0.0625], 'mean_feature_error_rates': 0.04513888888888889}
+```
+
+Error rates may be greater than 1.0 if the reference string is shorter than the prediction string:
+```python
+>>> phone_distance.compute(predictions=["bob"], references=["po"])
+{'phone_error_rates': [1.0], 'mean_phone_error_rate': 1.0, 'phone_feature_error_rates': [1.0416666666666667], 'mean_phone_feature_error_rates': 1.0416666666666667, 'feature_error_rates': [0.020833333333333332], 'mean_feature_error_rates': 0.020833333333333332}
+```
+
+Empty reference strings will cause an ValueError, you should handle them separately:
+```python
+>>> phone_distance.compute(predictions=["bob"], references=[""])
+Traceback (most recent call last):
+  ...
+    raise ValueError("one or more references are empty strings")
+ValueError: one or more references are empty strings
+```
 
 ## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
+- Phone error rate and feature error rate can be greater than 1.0 if the reference string is shorter than the prediction string.
+- Since these are error rates, not edit distances, the reference strings cannot be empty. 
 
 ## Citation
-*Cite the source where this metric was introduced.*
+```bibtex
+@inproceedings{Mortensen-et-al:2016,
+  author    = {David R. Mortensen and
+               Patrick Littell and
+               Akash Bharadwaj and
+               Kartik Goyal and
+               Chris Dyer and
+               Lori S. Levin},
+  title     = {PanPhon: {A} Resource for Mapping {IPA} Segments to Articulatory Feature Vectors},
+  booktitle = {Proceedings of {COLING} 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
+  pages     = {3475--3484},
+  publisher = {{ACL}},
+  year      = {2016}
+}
+```
 
 ## Further References
-*Add any useful further references.*
+- PER and PFER are used as evaluation metrics in [Universal Automatic Phonetic Transcription into the International Phonetic Alphabet (Taguchi et al.)](https://www.isca-archive.org/interspeech_2023/taguchi23_interspeech.html) 
+- Pierce Darragh's blog post [Introduction to Phonology, Part 3: Phonetic Features](https://pdarragh.github.io/blog/2018/04/26/intro-to-phonology-pt-3/) gives an overview of phonetic features for speech sounds.
+- [panphon Github repository](https://github.com/dmort27/panphon)

phone_distance.py

@@ -57,10 +57,9 @@ equality, but can also be normalized by number of phones.
 Each measure is given for each prediction, reference pair along with the mean value across all pairs. 
 
 Args:
-    predictions: list of predictions to score. Each predictions
-        should be a string of unicode characters.
-    references: list of reference for each prediction. Each
-        reference should be a string with of unicode characters.
+    predictions: list of predictions to score. Each predictions should be a string of unicode characters.
+    references: list of reference for each prediction. Each reference should be a string with of unicode characters.
+    feature_model: string to set which panphon.distance.Distance feature parsing model is used, choose from "strict", "permissive", "segment". Defaults to "segment".
     is_normalize_pfer: bool, set to True to normalize PFER by the largest number of phones in the prediction, reference pair
 Returns:
     phone_error_rates: list of floats giving PER for each prediction, reference pair
@@ -137,12 +136,12 @@ class PhoneDistance(evaluate.Metric):
             reference_urls=["https://pypi.org/project/panphon/", "https://arxiv.org/abs/2308.03917"]
         )
 
-    def _compute(self, predictions:list[str]|None=None, references:list[str]|None=None, feature_model:str="segment", is_normalize_pfer:bool=False):
+    def _compute(self, predictions:list[str], references:list[str], feature_model:str="segment", is_normalize_pfer:bool=False):
         """Computes phoneme error rates, phone feature error rate (Hamming feature edit distance) and feature error rates between prediction and reference strings 
 
         Args:
-            predictions (list[str], optional): Predicted transcriptions. Defaults to None.
-            references (list[str], optional): Reference transcriptions. Defaults to None.
+            predictions (list[str]): Predicted transcriptions.
+            references (list[str]): Reference transcriptions.
             feature_model (str, optional): panphon.distance.Distance feature parsing model to be used, choose from "strict", "permissive", "segment". Defaults to "segment".
             is_normalize_pfer (bool, optional): Set to true to normalize phone feature error rates by maximum length (measure won't be a true metric). Defaults to False.