Issue with `compute()` Method in Custom Hugging Face Metric Evaluation
Hello everyone!I’m working on creating a custom evaluation metric using the Hugging Faceevaluatelibrary, following this guide:huggingface.coCreating and sharing a new evaluationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.My custom metric doesn’t rely on the defaultpredictionsandreferencesparameters, but thecompute()method from theEvaluationModuleclass seems to require them to be passed, even though they are irrelevant for my use case.Here’s the method signature for reference:def compute(self, *, predictions=None, references=None, **kwargs) -> Optional[dict]:The issue I’m facing is that callingcompute()withoutpredictionsandreferencesleads to errors because the method expects these arguments to be added (viaadd_batchoradd) before callingcompute(). However, for my custom metric, these arguments are unnecessary.Does anyone know how I can work around this to usecompute()without needingpredictionsandreferences?Thanks in advance!
2025-01-01T23:12:00Z
How about creating a wrapper function that discards predictions and references, and specifying it from the Trainer?stackoverflow.comWhy do we need to write a function to "Compute Metrics" with Huggingface Question Answering Trainer when evaluating SQuAD?python, machine-learning, nlp, huggingface-transformersasked byignaciocton11:22AM - 15 Mar 23 UTC
2025-01-01T23:35:50Z
Thanks for the response, but it’s not quite what I’m looking for.I want to add a new metric to Hugging Face Spaces so it can be accessible to everyone. The metric is designed to evaluate models post-training.However, my metric doesn’t use the predictions and references parameters in the compute method, as shown in the guide. I’m asking if there’s a solution to this issue.Thanks.
2025-01-01T23:54:48Z
Hmm, so we really need to inherit from Evaluate. It’s difficult to change the arguments themselves in this case, so I can only think of receiving the arguments as a dummy and discarding the received arguments.huggingface.coCreating and sharing a new evaluationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.def _compute(self, references, predictions):
    #em = sum([r==p for r, p in zip(references, predictions)])/len(references)
    em = True
    return {"exact_match": em}