src/about.py

from dataclasses import dataclass
from enum import Enum


@dataclass
class Task:
    benchmark: str
    metric: str
    col_name: str


# Select your tasks here
# ---------------------------------------------------
class Tasks(Enum):
    # task_key in the json file, metric_key in the json file, name to display in the leaderboard
    task0 = Task("main", "acc", "Overall")


NUM_FEWSHOT = 0  # Change with your few shot
# ---------------------------------------------------


# Your leaderboard name
TITLE = """<h1 align="center" id="space-title">👩‍🏫Invalsi Leaderboard
<img src="https://huggingface.co/spaces/Crisp-Unimib/INVALSIbenchmark/resolve/main/src/logo-crisp-eng-retina.png" height="800" width="150" style="vertical-align: middle;">
</h1>"""

# What does your leaderboard evaluate?
INTRODUCTION_TEXT = """
Welcome into <a href="https://crispresearch.it/"><b>CRISP Bicocca</b></a> Invalsi Leaderboard page! 
On this page you will find the Invalsi Leaderboard: The Invalsi tests are tests prepared by the Invalsi Institute, administered to primary and secondary school students to assess their knowledge about mathematics and the Italian language. 

We structured a benchmark for assessing Italian capabilities of both open and close source Large Language Models, using 10 Invalsi tests, including all different grades of difficulty. (Paper Link)
"""

# Which evaluations are you running? how can people reproduce what you have?
LLM_BENCHMARKS_TEXT = f"""
## INVALSI Benchmark 

### Data Collection
We have collected from public sources 58 unique tests, divided into 141 unique units, with 2114 questions and 2808 unique items. Some questions are subdivided into multiple items, each requiring a specific answer. 
The data has been gathered from the <a href="https://www.gestinv.it/Index.aspx"><b>Gestinv</b></a> database.
Questions' formatting is sometimes not adequately structured for LLM evaluation; for instance, it is sometimes impossible to automatically transcribe the questions into structured fields, necessitating further inspection of images and PDFs. For this reason, we also collected corresponding PDF files and images. Manual inspection was required to ensure accuracy. In cases where questions involved graphical elements, we modified them into an appropriate multiple-choice format. For example, if the task required a student to find and underscore a word, we reformulated the question to allow selection from multiple choices. Similarly, if the task involved drawing a line between two groups of concepts—a common task for younger students—we rephrased it to involve choosing the correct association from given options. Generally, we aimed to adapt the questions to a format that allows the model to select the correct answer from a pool of choices if it aligns with the original type of question.


### Dataset Characteristics

<ul>
 <li> We have selected <b>10 tests</b> comprising <b>31 unique units</b>, <b>409 questions</b>, and <b>625 items</b> from the above data. A test comprises <b>two or more different units</b>; each question can have more than one item to answer. The sample of tests has been chosen by manual inspection, aiming to include different grades and years and avoiding those with questions that require inspecting an image or contained questions that would be hard to reformulate for language model comprehension. <br> <br>
<table style="font-size: 18px; border-collapse: collapse; width: 60%; margin: auto; border: 1px solid black;">
    <caption style="caption-side: top; text-align: center; font-weight: bold;">
        Distribution of tests, questions and items by educational grade.
    </caption>
    <thead>
        <tr style="background-color: #f2f2f2;">
            <th style="border: 1px solid black; padding: 8px; text-align: left;">School Grade</th>
            <th style="border: 1px solid black; padding: 8px; text-align: left;"># Tests</th>
            <th style="border: 1px solid black; padding: 8px; text-align: left;"># Questions</th>
            <th style="border: 1px solid black; padding: 8px; text-align: left;"># Items</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2nd Grade (Primary School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">34</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">72</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">5th Grade (Primary School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">75</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">117</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">6th Grade (Middle School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">1</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">87</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">118</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">8th Grade (Middle School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">86</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">88</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">10th Grade (High School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">2</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">77</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">139</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">13th Grade (High School)</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">1</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">50</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">91</td>
        </tr>
    </tbody>
</table> 
</li>
<br> <br>
<li> Each question in the benchmark is labelled with the specific <b>lexical macro area</b> it aims to assess <br> <br>

<table style="font-size: 18px; border-collapse: collapse; width: 80%; margin: auto; border: 1px solid black;">
    <caption style="caption-side: top; text-align: center; font-weight: bold;">
        Distribution of questions by section and macro area.
    </caption>
    <thead>
        <tr style="background-color: #f2f2f2;">
            <th style="border: 1px solid black; padding: 8px; text-align: left;">Section</th>
            <th style="border: 1px solid black; padding: 8px; text-align: left; width: 3.5cm;">Macro Area</th>
            <th style="border: 1px solid black; padding: 8px; text-align: right;"># Questions</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td rowspan="3" style="border: 1px solid black; padding: 8px; text-align: left; width: 1.3cm;">Text comprehension</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Reconstruct the meaning of the text, locally or globally</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">179 (43.8%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Locate and identify information within the text</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">108 (26.4%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Reflect on the content or form of the text, <br> locally or globally, and evaluate them</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">33 (8.1%)</td>
        </tr>
        <tr>
            <td rowspan="6" style="border: 1px solid black; padding: 8px; text-align: left; width: 1.3cm;">Reflection on the language</td>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Lexicon and semantics</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">29 (7.1%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Morphology</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">24 (5.9%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Syntax</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">19 (4.6%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Word formation</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">7 (1.7%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Textuality and pragmatics</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">6 (1.5%)</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 8px; text-align: left;">Spelling</td>
            <td style="border: 1px solid black; padding: 8px; text-align: right;">4 (1.0%)</td>
        </tr>
    </tbody>
</table>

</li>
<br> <br>

<li> The questions come in five distinct formats <br> <br> 
<table style="width: 90%; margin: auto; border-collapse: collapse; font-size: 18px; border: 1px solid black;">
    <caption style="caption-side: top; text-align: center; font-weight: bold;">
        Combined Legend of Question Types and Distribution by Format
    </caption>
    <thead>
        <tr style="background-color: #f2f2f2;">
            <th style="border: 1px solid black; padding: 10px; text-align: center;">Question Type</th>
            <th style="border: 1px solid black; padding: 10px; text-align: center;">Description</th>
            <th style="border: 1px solid black; padding: 10px; text-align: center;"># Questions</th>
            <th style="border: 1px solid black; padding: 10px; text-align: center;"># Items</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Multiple Choice (MC)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Composed of a question with several answer options, among which only one is correct. The most common question format.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">334 (81.7%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">340</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Multiple Complex Choice (MCC)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Composed of input questions and multiple items to answer. Requires all items to be correctly answered.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">38 (9.3%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">228</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Unique Response (RU)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Open-ended questions with only one correct answer, sometimes allowing limited variants.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">33 (8.1%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">50</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Short Response (RB)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Specific sub-type of RU questions requiring a single identifying word as the answer.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">1 (0.2%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">1</td>
        </tr>
        <tr>
            <td style="border: 1px solid black; padding: 10px; text-align: left;"><i>Cloze (CL)</i></td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">Text with missing words to be filled in, can be open-ended or closed-ended.</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">3 (0.7%)</td>
            <td style="border: 1px solid black; padding: 10px; text-align: left;">6</td>
        </tr>
    </tbody>
</table>
</il>

</ul>


"""

EVALUATION_QUEUE_TEXT = """
## Evaluation Form

Please submit your model for evaluation by filling out the form located [here](https://forms.gle/tkbJME4SS9LF8XTK7).

## Some good practices before submitting a model

### 1) Make sure you can load your model and tokenizer using AutoClasses:
```python
from transformers import AutoConfig, AutoModel, AutoTokenizer
config = AutoConfig.from_pretrained("your model name", revision=revision)
model = AutoModel.from_pretrained("your model name", revision=revision)
tokenizer = AutoTokenizer.from_pretrained("your model name", revision=revision)
```
If this step fails, follow the error messages to debug your model before submitting it. It's likely your model has been improperly uploaded.

Note: make sure your model is public!
Note: if your model needs `use_remote_code=True`, we do not support this option yet but we are working on adding it, stay posted!

### 2) Convert your model weights to [safetensors](https://huggingface.co/docs/safetensors/index)
It's a new format for storing weights which is safer and faster to load and use. It will also allow us to add the number of parameters of your model to the `Extended Viewer`!

### 3) Make sure your model has an open license!
This is a leaderboard for Open LLMs, and we'd love for as many people as possible to know they can use your model 🤗
"""

CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results"
CITATION_BUTTON_TEXT = r"""
"""