
# rugpt3xl
Model was trained with 512 sequence length using [Deepspeed](https://github.com/microsoft/DeepSpeed) and [Megatron](https://github.com/NVIDIA/Megatron-LM) code by [SberDevices](https://sberdevices.ru/) team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.  
*Note! Model has sparse attention blocks.*

Total training time was around 10 days on 256 GPUs.  
Final perplexity on test set is `12.05`.
Model parameters: 1.3B.

{"language": ["ru"], "tags": ["PyTorch", "Transformers"], "thumbnail": "https://github.com/sberbank-ai/ru-gpts"}
PyTorch Transformers ru region:us
TAGS
#PyTorch #Transformers #ru #region-us 


# rugpt3xl
Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.  
*Note! Model has sparse attention blocks.*

Total training time was around 10 days on 256 GPUs.  
Final perplexity on test set is '12.05'.
Model parameters: 1.3B.

# rugpt3xl
Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.  
*Note! Model has sparse attention blocks.*

Total training time was around 10 days on 256 GPUs.  
Final perplexity on test set is '12.05'.
Model parameters: 1.3B.
TAGS
#PyTorch #Transformers #ru #region-us 
 # rugpt3xl
Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.  
*Note! Model has sparse attention blocks.*

Total training time was around 10 days on 256 GPUs.  
Final perplexity on test set is '12.05'.
Model parameters: 1.3B.
passage: TAGS
#PyTorch #Transformers #ru #region-us 
# rugpt3xl
Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.  
*Note! Model has sparse attention blocks.*

Total training time was around 10 days on 256 GPUs.  
Final perplexity on test set is '12.05'.
Model parameters: 1.3B.