Just for performing some experiments. Do not use.

Since the loss seem to start going up, I did have to restore this from 9e945cb0636bde60bec30bd7df5db30f80401cc7 (2 step 600k/200). I am then restarting with warmup decaying from 1e-4.

That did failed. Checked out c94b5bb43b05fc798f9db013d940b05b3b47cd98 instead and restarted step 3 from here.



{}
transformers pytorch jax tensorboard roberta fill-mask autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #jax #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

Just for performing some experiments. Do not use.

Since the loss seem to start going up, I did have to restore this from 9e945cb0636bde60bec30bd7df5db30f80401cc7 (2 step 600k/200). I am then restarting with warmup decaying from 1e-4.

That did failed. Checked out c94b5bb43b05fc798f9db013d940b05b3b47cd98 instead and restarted step 3 from here.



TAGS
#transformers #pytorch #jax #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #jax #tensorboard #roberta #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
