DeBERTa trained from scratch

Source data: https://dumps.wikimedia.org/archive/2006/

Tools used: https://github.com/mikesong724/Point-in-Time-Language-Model

2006 wiki archive 2.7 GB             trained 24 epochs = 65GB

GLUE benchmark

	cola (3e): matthews corr: 0.2848
	
	sst2 (3e): acc: 0.8876
	
	mrpc (5e): F1: 0.8033, acc: 0.7108
	
	stsb (3e): pearson: 0.7542, spearman: 0.7536
	
	qqp (3e): acc: 0.8852, F1: 0.8461
	
	mnli (3e): acc_mm: 0.7822
	
	qnli (3e): acc: 0.8715
	
	rte (3e): acc: 0.5235											
	
	wnli (5e): acc: 0.3099
{}
transformers pytorch deberta fill-mask autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #deberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

DeBERTa trained from scratch

Source data: URL

Tools used: URL

2006 wiki archive 2.7 GB             trained 24 epochs = 65GB

GLUE benchmark

	cola (3e): matthews corr: 0.2848
	
	sst2 (3e): acc: 0.8876
	
	mrpc (5e): F1: 0.8033, acc: 0.7108
	
	stsb (3e): pearson: 0.7542, spearman: 0.7536
	
	qqp (3e): acc: 0.8852, F1: 0.8461
	
	mnli (3e): acc_mm: 0.7822
	
	qnli (3e): acc: 0.8715
	
	rte (3e): acc: 0.5235											
	
	wnli (5e): acc: 0.3099
TAGS
#transformers #pytorch #deberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #deberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
