w2v-bert-2.0-lg-CV-Fleurs-10hrs-v10

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.8037
• Wer: 0.3687
• Cer: 0.0789

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
0.92	1.0	323	0.4679	0.4702	0.1007
0.3291	2.0	646	0.3830	0.4235	0.0888
0.2652	3.0	969	0.3788	0.4045	0.0902
0.2197	4.0	1292	0.3715	0.3921	0.0856
0.1859	5.0	1615	0.3861	0.3773	0.0815
0.16	6.0	1938	0.3732	0.3857	0.0806
0.1352	7.0	2261	0.3669	0.3798	0.0817
0.118	8.0	2584	0.4030	0.3708	0.0798
0.101	9.0	2907	0.4224	0.3648	0.0774
0.087	10.0	3230	0.4248	0.3702	0.0785
0.0728	11.0	3553	0.4548	0.3680	0.0786
0.0622	12.0	3876	0.4699	0.3776	0.0809
0.0516	13.0	4199	0.5133	0.3711	0.0791
0.0446	14.0	4522	0.4872	0.3871	0.0822
0.0373	15.0	4845	0.5609	0.3636	0.0781
0.0346	16.0	5168	0.6009	0.3851	0.0793
0.0321	17.0	5491	0.5736	0.3720	0.0798
0.0275	18.0	5814	0.6151	0.3802	0.0813
0.0239	19.0	6137	0.6704	0.3751	0.0790
0.0227	20.0	6460	0.6345	0.3729	0.0788
0.0193	21.0	6783	0.5992	0.3791	0.0811
0.0173	22.0	7106	0.6686	0.3844	0.0808
0.016	23.0	7429	0.6537	0.3679	0.0785
0.0157	24.0	7752	0.6662	0.3791	0.0799
0.014	25.0	8075	0.6934	0.3596	0.0777
0.0142	26.0	8398	0.6658	0.3545	0.0763
0.0128	27.0	8721	0.6774	0.3636	0.0782
0.0123	28.0	9044	0.6936	0.3723	0.0804
0.012	29.0	9367	0.6774	0.3597	0.0781
0.0103	30.0	9690	0.7382	0.3524	0.0763
0.0094	31.0	10013	0.7149	0.3582	0.0775
0.0103	32.0	10336	0.7377	0.3739	0.0795
0.0087	33.0	10659	0.7493	0.3515	0.0760
0.0098	34.0	10982	0.7361	0.3581	0.0770
0.0078	35.0	11305	0.7041	0.3705	0.0782
0.0071	36.0	11628	0.7752	0.3563	0.0767
0.0085	37.0	11951	0.7123	0.3612	0.0783
0.0073	38.0	12274	0.7649	0.3716	0.0782
0.0063	39.0	12597	0.7724	0.3559	0.0765
0.0072	40.0	12920	0.7587	0.3626	0.0784
0.0065	41.0	13243	0.7694	0.3794	0.0792
0.0063	42.0	13566	0.8037	0.3687	0.0789

Framework versions

• Transformers 4.46.3
• Pytorch 2.1.0+cu118
• Datasets 3.1.0
• Tokenizers 0.20.3