Updates README

Signed-off-by: Giovani <[email protected]>

README.md

@@ -11,11 +11,9 @@ metrics:
 
 <!-- Provide a quick summary of what the model is/does. -->
 
-This is a XLM-RoBERTa-base fine-tuned model on 5K (premise, hypothesis) sentence pairs from
-the ASSIN (Avaliação de Similaridade Semântica e Inferência textual) corpus. Both the original corpus
-and XLM-RoBERTa-base model can be found here. The original reference papers are:
-Unsupervised Cross-Lingual Representation Learning At Scale, ASSIN: Avaliação de Similaridade Semântica e 
-Inferência Textual, respectivelly. This model is suitable for Portuguese (from Brazil or Portugal).
+This is a **[XLM-RoBERTa-base](https://huggingface.co/xlm-roberta-base) fine-tuned model** on 5K (premise, hypothesis) sentence pairs from
+the **ASSIN (Avaliação de Similaridade Semântica e Inferência textual)** corpus. The original reference papers are:
+[Unsupervised Cross-Lingual Representation Learning At Scale](https://arxiv.org/pdf/1911.02116), [ASSIN: Avaliação de Similaridade Semântica e Inferência Textual](https://huggingface.co/datasets/assin), respectivelly. This model is suitable for Portuguese (from Brazil or Portugal).
 
 ## Model Details
 
@@ -23,13 +21,12 @@ Inferência Textual, respectivelly. This model is suitable for Portuguese (from
 
 <!-- Provide a longer summary of what this model is. -->
 
-
-
 - **Developed by:** Giovani Tavares and Felipe Ribas Serras
+- **Oriented By:** Felipe Ribas Serras, Renata Wassermann and Marcelo Finger
 - **Shared by [optional]:** [More Information Needed]
 - **Model type:** Transformer-based text classifier
 - **Language(s) (NLP):** Portuguese
-- **License:** [More Information Needed]
+- **License:** mit
 - **Finetuned from model [optional]:** [XLM-RoBERTa-base](https://huggingface.co/xlm-roberta-base)
 
 ### Model Sources [optional]
@@ -46,113 +43,143 @@ Inferência Textual, respectivelly. This model is suitable for Portuguese (from
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
+This fine-tuned version of [XLM-RoBERTa-base](https://huggingface.co/xlm-roberta-base) performs Natural
+Language Inference (NLI), which is a text classification task.
 
-### Downstream Use [optional]
+<div id="assin_function">
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+**Definition 1.** Given a pair of sentences $(premise, hypothesis)$, let $\hat{f}^{(xlmr\_base)}$ be the fine-tuned models' inference function:
 
-[More Information Needed]
+$$
+\hat{f}^{(xlmr\_base)} = 
+\begin{cases} 
+ENTAILMENT, & \text{if $premise$ entails $hypothesis$}\\
+PARAPHRASE, & \text{if $premise$ entails $hypothesis$ and $hypothesis$ entails $premise$}\\
+NONE & \text{otherwise}
+\end{cases}
+$$
+</div>
 
-### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+The $(premise, hypothesis)$ entailment definition used is the same as the one found in Salvatore's paper [1].
 
-[More Information Needed]
-
-## Bias, Risks, and Limitations
+Therefore, **this fine-tuned version of [XLM-RoBERTa-base](https://huggingface.co/xlm-roberta-base) classifies pairs of sentences into one of the following classes $ENTAILMENT, PARAPHRASE$ or $NONE$.** using [Definition 1](#assin_function).
+ 
+<!-- ## Bias, Risks, and Limitations
 
 <!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-[More Information Needed]
-
 ### Recommendations
 
 <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+This model should be used for scientific purposes only. It was not tested for production environments.
 
-## How to Get Started with the Model
+<!-- ## How to Get Started with the Model
 
 Use the code below to get started with the model.
 
-[More Information Needed]
+[More Information Needed] -->
 
 ## Fine-Tuning Details
 
 ### Fine-Tuning Data
 
 <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+---
 
-This is a fine tuned version of [XLM-RoBERTa-base](https://huggingface.co/xlm-roberta-base) using the [ASSIN (Avaliação de Similaridade Semântica e Inferência textual)](https://huggingface.co/datasets/assin)
-[More Information Needed] dataset. [ASSIN](https://huggingface.co/datasets/assin) is a corpus annotated with hypothesis/premise Portuguese sentence pairs suitable for detecting textual entailment, paraphrase or neutral 
+- **Train Dataset**: [ASSIN](https://huggingface.co/datasets/assin) <br>
+
+- **Evaluation Dataset used for Hyperparameter Tuning:** [ASSIN](https://huggingface.co/datasets/assin)'s validation split
+
+- **Test Datasets:**
+    - [ASSIN](https://huggingface.co/datasets/assin)'s test splits
+    - [ASSIN2](https://huggingface.co/datasets/assin2)'s test splits
+
+
+---
+This is a fine tuned version of [XLM-RoBERTa-base](https://huggingface.co/xlm-roberta-base) using the [ASSIN (Avaliação de Similaridade Semântica e Inferência textual)](https://huggingface.co/datasets/assin) dataset. [ASSIN](https://huggingface.co/datasets/assin) is a corpus annotated with hypothesis/premise Portuguese sentence pairs suitable for detecting textual entailment, paraphrase or neutral 
 relationship between the members of such pairs. Such corpus has three subsets: *ptbr* (Brazilian Portuguese), *ptpt* (Portuguese Portuguese) and *full* (the union of the latter with the former). The *full* subset has 
-$10k$ sentence pairs equally distributed between *ptbr*  and *ptpt* subsets.
-[More Information Needed]
+10k sentence pairs equally distributed between *ptbr*  and *ptpt* subsets.
 
 ### Fine-Tuning Procedure 
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-The model fine-tuning procedure can be summarized in three major subsequent tasks:
+The model's fine-tuning procedure can be summarized in three major subsequent tasks:
     <ol type="i">
-        <li>**[Data Processing](#data-processing):**</li> [ASSIN](https://huggingface.co/datasets/assin)'s *validation* and *train* splits were loaded from the **Hugging Face Hub** and processed afterwards; 
+        <li>**Data Processing:**</li> [ASSIN](https://huggingface.co/datasets/assin)'s *validation* and *train* splits were loaded from the **Hugging Face Hub** and processed afterwards; 
         <li>**Hyperparameter Tuning:**</li> [XLM-RoBERTa-base](https://huggingface.co/xlm-roberta-base)'s hyperparameters were chosen with the help of the [Weights & Biases] API to track the results and upload the fine-tuned models;
         <li>**Final Model Loading and Testing:**</li>
         using the *cross-tests* approach described in the [this section](#evaluation), the models' performance were measured using different datasets and metrics. 
     </ol>
-#### Data Processing [optional]
-##### Class Label Column Renaming
-The **Hugging Face**'s ```transformers``` module's ```DataCollator``` used by its ```Trainer``` requires that the ```class label``` column of the collated dataset to be called ```label```.  [ASSIN](https://huggingface.co/datasets/assin)'s class label column for each hypothesis/premise pair is called ```entailment_judgement```. Therefore, as the first step of the data preprocessing pipeline the column  ```entailment_judgement``` was renamed to ```label``` so that the **Hugging Face**'s ```transformers``` module's ```Trainer``` could be used.
 
-#### Training Hyperparameters
+More information on the fine-tuning procedure can be found in [@tcc_paper].
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
-#### Speeds, Sizes, Times [optional]
+<!--  ##### Column Renaming
+The **Hugging Face**'s ```transformers``` module's ```DataCollator``` used by its ```Trainer``` requires that the ```class label``` column of the collated dataset to be called ```label```.  [ASSIN](https://huggingface.co/datasets/assin)'s class label column for each hypothesis/premise pair is called ```entailment_judgement```. Therefore, as the first step of the data preprocessing pipeline the column  ```entailment_judgement``` was renamed to ```label``` so that the **Hugging Face**'s ```transformers``` module's ```Trainer``` could be used. -->
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+#### Hyperparameter Tuning
 
-[More Information Needed]
+The model's training hyperparameters were chosen according to the following definition:
 
-## Evaluation
+<div id="hyperparameter_tuning">
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+**Definition 2.** Let $Hyperparms= \{i: i \text{ is an hyperparameter of } \hat{f}^{(xlmr\_base)}\}$ and $\hat{f}^{(xlmr\_base)}$ be the model's inference function defined in [Definition 1](#assin_function) :
 
-### Testing Data, Factors & Metrics
+$$
+Hyperparms = \argmax_{hyp}(eval\_acc(\hat{f}^{(xlmr\_base)}_{hyp}, assin\_validation))
+$$
+</div>
 
-#### Testing Data
+The following hyperparameters were tested in order to maximize the evaluation accuracy.
 
-<!-- This should link to a Data Card if possible. -->
+- **Number of Training Epochs:** $(1,2,3)$
+- **Per Device Train Batch Size:** $(16,32)$
+- **Learning Rate:** $(1e-6, 2e-6,3e-6)$
 
-[More Information Needed]
 
-#### Factors
+The hyperaparemeter tuning experiments were run and tracked using the [Weights & Biases' API](https://docs.wandb.ai/ref/python/public-api/api) and can be found at this [link](https://wandb.ai/gio_projs/assin_xlm_roberta_v5?workspace=user-giogvn).
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
 
-[More Information Needed]
+#### Training Hyperparameters
 
-#### Metrics
+The [hyperparameter tuning](#hyperparameter-tuning) performed yelded the following values:
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+- **Number of Training Epochs:** $3$
+- **Per Device Train Batch Size:** $16$
+- **Learning Rate:** $3e-6$
+
+## Evaluation
 
-[More Information Needed]
+### ASSIN
 
-### Results
+Testing this model in [ASSIN](https://huggingface.co/datasets/assin)'s test split is straightforward. The following code snippet shows how to do it:
 
-[More Information Needed]
+### ASSIN2
 
-#### Summary
+Given a pair of sentences $(premise, hypothesis)$, $\hat{f}^{(xlmr\_base)}(premise, hypothesis)$ can be equal to $PARAPHRASE, ENTAILMENT$ or $NONE$ as defined in [Definition 1](#assin_function). 
 
+[ASSIN2](https://huggingface.co/datasets/assin2)'s test split's class label's column has only two possible values: $ENTAILMENT$ and $NONE$. Therefore, in order to test this model in [ASSIN2](https://huggingface.co/datasets/assin2)'s test split some mapping must be done in order to make the [ASSIN2](https://huggingface.co/datasets/assin2)' class labels compatible with the model's inference function.
 
+More information on how such mapping is performed can be found in [Modelos para Inferência em Linguagem Natural que entendem a Língua Portuguesa](https://linux.ime.usp.br/~giovani/).
 
-## Model Examination [optional]
+### Metrics
 
-<!-- Relevant interpretability work for the model goes here -->
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+The model's performance metrics for each test dataset are presented separately. Accuracy, f1 score, precision and recall were the metrics used to every evaluation performed. Such metrics are reported below. More information on such metrics them can be found in [2].
 
-[More Information Needed]
+### Results
+
+| test set | accuracy | f1 score | precision | recall |
+|----------|----------|----------|-----------|--------|
+| assin    |0.89      |0.89      |0.89       |0.89    |
+| assin2   |0.70      |0.69      |0.73       |0.70    |
+
+## Model Examination
+
+<!-- Relevant interpretability work for the model goes here -->
+Some interpretability work was done in order to understand the model's behavior. Such work can be found in the paper describing the procedure to create this fine-tuned model in [@tcc_paper].
 
 ## Environmental Impact
 
@@ -166,50 +193,25 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 - **Compute Region:** [More Information Needed]
 - **Carbon Emitted:** [More Information Needed]
 
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
+## Citation
 
 <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
 **BibTeX:**
 
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
+```bibtex
+    @article{tcc_paper,
+    author    = {Giovani Tavares and Felipe Ribas Serras and Renata Wassermann and Marcelo Finger},
+    title     = {Modelos Transformer para Inferência de Linguagem Natural em Português},
+    pages     = {x--y},
+    year      = {2023}
+    }
+```
 
-## Model Card Authors [optional]
+## References
 
-[More Information Needed]
+[1][Salvatore, F. S. (2020). Analyzing Natural Language Inference from a Rigorous Point of View (pp. 1-2).](https://www.teses.usp.br/teses/disponiveis/45/45134/tde-05012021-151600/publico/tese_de_doutorado_felipe_salvatore.pdf)
 
-## Model Card Contact
+[2][Andrade, G. T. (2023)  Modelos para Inferência em Linguagem Natural que entendem a Língua Portuguesa  (train_assin_xlmr_base_results PAGES GO HERE)](https://linux.ime.usp.br/~giovani/)
 
-[More Information Needed]
+[3][Andrade, G. T. (2023)  Modelos para Inferência em Linguagem Natural que entendem a Língua Portuguesa (train_assin_xlmr_base_conclusions PAGES GO HERE)](https://linux.ime.usp.br/~giovani/)