You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Global Batch size : 384 seq_len: 2048

Checkpoint every 500 steps

i.e every 393216000 tokens or 400M Tokens

Current Revison available as

  • checkpoint-500 393M
  • checkpoint-1000 786M
  • checkpoint-1500 1.18B
  • checkpoint-2000 1.57B
  • checkpoint-2500 1.96B

max_lr : 7e-5

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no pipeline_tag.