|
--- |
|
license: bigscience-bloom-rail-1.0 |
|
datasets: |
|
- swap-uniba/itwiki-march-2024 |
|
language: |
|
- it |
|
tags: |
|
- bloom |
|
- italian |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
The model is obtained by performing language adaptation on the original bloom-1b7 model. |
|
In detail, we continued the pre-training on Italian-specific data without adaptation of the vocabulary. |
|
We use about 2.8M documents obtained from Italian Wikimedia dumps (swap-uniba/itwiki-march-2024). The model is trained for one epoch using LoRA and SFT. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **Developed by:** SWAP Research Group, Department of Computer Science, University of Bari Aldo Moro |
|
- **Model type:** BLOOM |
|
- **Language(s) (NLP):** Italian |
|
- **License:** bigscience-bloom-rail-1.0 |
|
- **Finetuned from model [optional]:** bloom-1b7 |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
2.8M documents obtained from Italian Wikimedia dumps (swap-uniba/itwiki-march-2024). |
|
|
|
### Training Procedure |
|
|
|
LoRA and SFT. |
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** fp16 |
|
|
|
## Citation [optional] |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
**BibTeX:** |
|
|
|
**APA:** |
|
|
|
## Model Card Authors [optional] |
|
|
|
Pierpaolo Basile, University of Bari Aldo Moro, Italy. |
|
|
|
## Model Card Contact |
|
|
|
Pierpaolo Basile, University of Bari Aldo Moro, Italy. |
