
DistilCamemBERT-QA
==================

We present DistilCamemBERT-QA, which is [DistilCamemBERT](https://huggingface.co/cmarkea/distilcamembert-base) fine-tuned for the Question-Answering task for the french language. This model is built using two datasets, FQuAD v1.0 and Piaf, composed of contexts and questions with their answers inside the context.

This modelization is close to [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) based on [CamemBERT](https://huggingface.co/camembert-base) model. The problem of the modelizations based on CamemBERT is at the scaling moment, for the production phase, for example. Indeed, inference cost can be a technological issue, especially in cross-encoding like this task. To counteract this effect, we propose this modelization which divides the inference time by 2 with the same consumption power, thanks to DistilCamemBERT.

Dataset
-------

The dataset comprises FQuAD v1.0 and Piaf with 24'566 questions and answers for the training set and 3'188 for the evaluation set.

Evaluation results and benchmark
--------------------------------

We compare [DistilCamemBERT-QA](https://huggingface.co/cmarkea/distilcamembert-base-qa) to two other modelizations working on the french language. The first one [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) is based on well named [CamemBERT](https://huggingface.co/camembert-base), the french RoBERTa model and the second one [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) is based on [FlauBERT](https://huggingface.co/flaubert/flaubert_base_uncased) another french model based on BERT architecture this time. 

For our benchmarks, we do a word-to-word comparison between words that are matching between the predicted answer and the ground truth. We also use f1-score, which measures the intersection quality between predicted responses and ground truth. Finally, we use inclusion score, which measures if the ground truth answer is included in the predicted answer. An **AMD Ryzen 5 4500U @ 2.3GHz with 6 cores** was used for the mean inference time measure.

| **model**          | **time (ms)** | **exact match (%)** | **f1-score (%)** | **inclusion-score (%)** |
| :--------------: | :-----------: | :--------------: | :------------: | :------------: |
| [cmarkea/distilcamembert-base-qa](https://huggingface.co/cmarkea/distilcamembert-base-qa) | **216.96** | 25.66 | 62.65 | 59.82 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 432.17 | **59.76** | **79.57** | **69.23** |
| [fmikaelian/flaubert-base-uncased-squad](https://huggingface.co/fmikaelian/flaubert-base-uncased-squad) | 875.84 | 0.22   | 5.21 | 3.68 |

Do not take into account the results of the FlauBERT model. The modeling seems to be a problem, as the results seem very low.

How to use DistilCamemBERT-QA
------------------------------

```python
from transformers import pipeline

qa_engine = pipeline(
    "question-answering",
    model="cmarkea/distilcamembert-base-qa",
    tokenizer="cmarkea/distilcamembert-base-qa"
)

result = qa_engine(
    context="David Fincher, né le 28 août 1962 à Denver (Colorado), "
    "est un réalisateur et producteur américain. Il est principalement "
    "connu pour avoir réalisé les films Seven, Fight Club, L'Étrange "
    "Histoire de Benjamin Button, The Social Network et Gone Girl qui "
    "lui ont valu diverses récompenses et nominations aux Oscars du "
    "cinéma ou aux Golden Globes. Réputé pour son perfectionnisme, il "
    "peut tourner un très grand nombre de prises de ses plans et "
    "séquences afin d'obtenir le rendu visuel qu'il désire. Il a "
    "également développé et produit les séries télévisées House of "
    "Cards (pour laquelle il remporte l'Emmy Award de la meilleure "
    "réalisation pour une série dramatique en 2013) et Mindhunter, "
    "diffusées sur Netflix.",
    question="Quel est le métier de David Fincher ?"
)

result
{'score': 0.7981914281845093,
 'start': 61,
 'end': 98,
 'answer': ' réalisateur et producteur américain.'}
```

### Optimum + ONNX

```python
from optimum.onnxruntime import ORTModelForQuestionAnswering
from transformers import AutoTokenizer, pipeline

HUB_MODEL = "cmarkea/distilcamembert-base-qa"

tokenizer = AutoTokenizer.from_pretrained(HUB_MODEL)
model = ORTModelForQuestionAnswering.from_pretrained(HUB_MODEL)
onnx_qa = pipeline("question-answering", model=model, tokenizer=tokenizer)

# Quantized onnx model
quantized_model = ORTModelForQuestionAnswering.from_pretrained(
    HUB_MODEL, file_name="model_quantized.onnx"
)
```

Citation
--------
```bibtex
@inproceedings{delestre:hal-03674695,
  TITLE = {{DistilCamemBERT : une distillation du mod{\`e}le fran{\c c}ais CamemBERT}},
  AUTHOR = {Delestre, Cyrile and Amar, Abibatou},
  URL = {https://hal.archives-ouvertes.fr/hal-03674695},
  BOOKTITLE = {{CAp (Conf{\'e}rence sur l'Apprentissage automatique)}},
  ADDRESS = {Vannes, France},
  YEAR = {2022},
  MONTH = Jul,
  KEYWORDS = {NLP ; Transformers ; CamemBERT ; Distillation},
  PDF = {https://hal.archives-ouvertes.fr/hal-03674695/file/cap2022.pdf},
  HAL_ID = {hal-03674695},
  HAL_VERSION = {v1},
}
```
{"language": "fr", "license": "cc-by-nc-sa-3.0", "datasets": ["fquad", "piaf"], "widget": [{"text": "Quand et o\u00f9 est sorti Toy Story ?", "context": "Pixar Animation Studios, ou simplement Pixar dans le langage courant, est une soci\u00e9t\u00e9 am\u00e9ricaine de production de films en images tridimensionnelles de synth\u00e8se. Elle a acquis sa notori\u00e9t\u00e9 gr\u00e2ce \u00e0 Toy Story, premier long m\u00e9trage de ce type, sorti aux \u00c9tats-Unis en 1995. \u00c0 ce jour, le studio d'animation a remport\u00e9 dix-neuf Oscars, quatre Golden Globes et trois Grammy Awards ainsi que de nombreuses autres r\u00e9compenses. Le studio travaille avec PhotoRealistic RenderMan, sa propre version de l'interface de programmation de rendu RenderMan utilis\u00e9e pour cr\u00e9er des images de haute qualit\u00e9. Ses studios de production et son si\u00e8ge social se trouvent au Pixar Campus situ\u00e9 \u00e0 Emeryville pr\u00e8s de San Francisco en Californie."}, {"text": "Quel est le premier long m\u00e9trage du studio ?", "context": "Pixar Animation Studios, ou simplement Pixar dans le langage courant, est une soci\u00e9t\u00e9 am\u00e9ricaine de production de films en images tridimensionnelles de synth\u00e8se. Elle a acquis sa notori\u00e9t\u00e9 gr\u00e2ce \u00e0 Toy Story, premier long m\u00e9trage de ce type, sorti aux \u00c9tats-Unis en 1995. \u00c0 ce jour, le studio d'animation a remport\u00e9 dix-neuf Oscars, quatre Golden Globes et trois Grammy Awards ainsi que de nombreuses autres r\u00e9compenses. Le studio travaille avec PhotoRealistic RenderMan, sa propre version de l'interface de programmation de rendu RenderMan utilis\u00e9e pour cr\u00e9er des images de haute qualit\u00e9. Ses studios de production et son si\u00e8ge social se trouvent au Pixar Campus situ\u00e9 \u00e0 Emeryville pr\u00e8s de San Francisco en Californie."}]}
transformers pytorch tf onnx safetensors camembert question-answering fr dataset:fquad dataset:piaf license:cc-by-nc-sa-3.0 endpoints_compatible region:us
TAGS
#transformers #pytorch #tf #onnx #safetensors #camembert #question-answering #fr #dataset-fquad #dataset-piaf #license-cc-by-nc-sa-3.0 #endpoints_compatible #region-us 

DistilCamemBERT-QA
==================


We present DistilCamemBERT-QA, which is DistilCamemBERT fine-tuned for the Question-Answering task for the french language. This model is built using two datasets, FQuAD v1.0 and Piaf, composed of contexts and questions with their answers inside the context.


This modelization is close to etalab-ia/camembert-base-squadFR-fquad-piaf based on CamemBERT model. The problem of the modelizations based on CamemBERT is at the scaling moment, for the production phase, for example. Indeed, inference cost can be a technological issue, especially in cross-encoding like this task. To counteract this effect, we propose this modelization which divides the inference time by 2 with the same consumption power, thanks to DistilCamemBERT.


Dataset
-------


The dataset comprises FQuAD v1.0 and Piaf with 24'566 questions and answers for the training set and 3'188 for the evaluation set.


Evaluation results and benchmark
--------------------------------


We compare DistilCamemBERT-QA to two other modelizations working on the french language. The first one etalab-ia/camembert-base-squadFR-fquad-piaf is based on well named CamemBERT, the french RoBERTa model and the second one fmikaelian/flaubert-base-uncased-squad is based on FlauBERT another french model based on BERT architecture this time.


For our benchmarks, we do a word-to-word comparison between words that are matching between the predicted answer and the ground truth. We also use f1-score, which measures the intersection quality between predicted responses and ground truth. Finally, we use inclusion score, which measures if the ground truth answer is included in the predicted answer. An AMD Ryzen 5 4500U @ 2.3GHz with 6 cores was used for the mean inference time measure.



Do not take into account the results of the FlauBERT model. The modeling seems to be a problem, as the results seem very low.


How to use DistilCamemBERT-QA
-----------------------------


### Optimum + ONNX


Citation
--------



### Optimum + ONNX


Citation
--------
TAGS
#transformers #pytorch #tf #onnx #safetensors #camembert #question-answering #fr #dataset-fquad #dataset-piaf #license-cc-by-nc-sa-3.0 #endpoints_compatible #region-us 
 ### Optimum + ONNX


Citation
--------
passage: TAGS
#transformers #pytorch #tf #onnx #safetensors #camembert #question-answering #fr #dataset-fquad #dataset-piaf #license-cc-by-nc-sa-3.0 #endpoints_compatible #region-us 
### Optimum + ONNX


Citation
--------