src/about.py

from dataclasses import dataclass
from enum import Enum
from src.envs import REPO_ID


@dataclass
class Task:
    benchmark: str
    metric: str
    col_name: str


# Select your tasks here
# ---------------------------------------------------
class Tasks(Enum):
    # task_key in the json file, metric_key in the json file, name to display in the leaderboard 
    task1 = Task("PeKA", "acc", "PeKA*")
    task2 = Task("PKBETS MCQA", "acc", "PKBETS MCQA*")
    task3 = Task("khayyam_challenge", "acc", "Khayyam Challenge")
    task4 = Task("parsinlu_mc", "acc", "ParsiNLU MCQA")
    task5 = Task("parsinlu_nli", "acc", "ParsiNLU NLI")
    task6 = Task("parsinlu_qqp", "acc", "ParsiNLU QQP")
    task7 = Task("persian_ARC", "acc", "Persian ARC-C")

NUM_FEWSHOT = 0 # Change with your few shot
# ---------------------------------------------------



# Your leaderboard name
TITLE = f"""
<img src="https://huggingface.co/spaces/{REPO_ID}/resolve/main/banner_green.png" style="width:70%;display:block;margin-left:auto;margin-right:auto">
"""


# What does your leaderboard evaluate?
INTRODUCTION_TEXT = """
Persian LLM Leaderboard is designed to be a challenging benchmark and provide a reliable evaluation of LLMs in Persian Language. 

Note: This is a demo version of the leaderboard. Two new benchmarks are introduced: *PeKA* and *PK-BETS*, challenging the native knowledge of the models along with 
linguistic skills and their level of bias, ethics, and trustworthiness. **These datasets are not yet public, but they will be uploaded onto huggingface along with a detailed paper
explaining the data and performance of relevent models.**

Note: **We plan to release an evaluation framework soon in which the details and methods of evaluation are specified.**
"""

# Which evaluations are you running? how can people reproduce what you have?
LLM_BENCHMARKS_TEXT = f"""
## ABOUT
For now, the only competitive open language models capable of properly speaking Persian are the multilingual ones, Meta's Llama 3.1 being the prime example. 
There are only a few capable multilingual LLMs in Persian that derive their main knowledge from English. A Persian LLM is almost an imagination right now as there doesn't exist
that many models being expert in Persian in the first place.

Our goal is to provide a benchmark on diverse domains and tasks that provide insights on how much is the gap between current Persian LLMs and the SOTA multilingual models right now in different grounds.
This benchmark can also be used by multilingual researchers to measure how well their model performs in a language like Persian.

We use our own framework to evaluate the models on the following benchmarks (TO BE RELEASED SOON).
### Tasks 
- PeKA: Persian Knowledge Assesment (0-shot) - a set of multiple-choice questions that tests the level of native knowledge in Persian language in more 15 domains and categories: From art to history and geography, cinema, tv, sports, law and medicine, and much more.
- PK-BETS: Persian Knowledge: Bias Ethics Toxicity and Skills (0-shot) - a test of model's knowledge in Persian and its capability in linguistic skills such as Grammar and Praphrasing, and also questions examining the bias, ethics, and toxicity of the model.
- <a href="https://arxiv.org/abs/2404.06644" target="_blank">  Khayyam Challenge (Persian MMLU) </a>  (0-shot) - comprising 20,805 four-choice questions (of which we use 20,776, removing questions that are longer than 200 words) sourced from 38 diverse tasks extracted from Persian examinations, spanning a wide spectrum of subjects, complexities, and ages
- <a href="https://arxiv.org/abs/2012.06154" target="_blank">  ParsiNLU MCQA </a> (0-shot) - a series of multiple-choice questions in domains of *literature*, *math & logic*, and *common knowledge*.
- <a href="https://arxiv.org/abs/2012.06154" target="_blank">  ParsiNLU NLI </a> (max[0,3,5,10]-shot) - a 3-way classification to determine whether a hypothesis sentence entails, contradicts, or is neutral with respect to a given premise sentence.
- <a href="https://arxiv.org/abs/2012.06154" target="_blank">  ParsiNLU QQP </a> (max[0,2,5,10]-shot) - task of deciding whether a whether two given questions are paraphrases of each other or not.
- <a href="https://huggingface.co/datasets/MatinaLLM/persian_arc" target="_blank">  Persian ARC-C</a> (0-shot) - <a href="https://huggingface.co/datasets/allenai/ai2_arc" target="_blank"> ARC (challenging subset) </a> dataset translated to Persian using GPT-4o.

For all these evaluations, a higher score is a better score.

We use the given *test* subset (for those benchmarks that also have *train* and *dev* subsets) for all these evaluations.

These benchmarks are picked for now, but several other benchmarks are going to be added later to help us perform a more thorough examination of models.

The benchmarks ParsiNLU NLI and ParsiNLU QQP are evaluated in different few-shot settings and then the maximum score is returned as the final evaluation.
We argue that this is indeed a fair evaluation scheme since many light-weight models (around ~7B and less) can have a poor in-context learning in long-context prompts and thus perform better
in smaller shots (or have a small knowledge capacity and perform poorly in zero-shot). We wish to not hold this against the model by trying to measure performances in different settings and take the maximum score achieved .

## REPRODUCIBILITY
The parameters used for evaluation along with instructions and prompts will be available once the framework is released. (TO BE COMPLETED)
"""

EVALUATION_QUEUE_TEXT = """

## Important Notes

- Right now, the models added **are not automatically evaluated**. - We may support automatic evaluation in the future on our own clusters.
- An evaluation framework will be available in the future to help everyone reproduce the results.
- We only support models with **a causal language modeling head** for now.

## Don't forget to read the FAQ and the About tabs for more information!

## First steps before submitting a model

### 1) Make sure you can load your model and tokenizer using AutoClasses:
```python
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
config = AutoConfig.from_pretrained("your model name", revision=revision)
model = AutoModelForCausalLM.from_pretrained("your model name", revision=revision)
tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)
```
If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded.

Note: make sure your model is public!

### 2) Make sure your model has an open license!
This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model 🤗

### 3) Fill up your model card
When we add extra information about models to the leaderboard, it will be automatically taken from the model card

### 4) Select the correct precision
Not all models are converted properly from `float16` to `bfloat16`, and selecting the wrong precision can sometimes cause evaluation error (as loading a `bf16` model in `fp16` can sometimes generate NaNs, depending on the weight range).


## In case of model failure
If your model is displayed in the `FAILED` category, its execution stopped.
Make sure you have followed the above steps first.

"""

CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
CITATION_BUTTON_TEXT = r"""
"""