allenai
/

Llama-3.1-Tulu-3-70B-SFT

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

vwxyzjn commited on 13 days ago

Commit

0c898d4

·

verified ·

1 Parent(s): 7eb5065

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -246,8 +246,8 @@ See the Falcon 180B model card for an example of this.
 ## Hyperparamters
 SFT:
-- **Learning Rate**: 5E-6 (8B), 2E-6 (70B)
-- **Effective Batch Size:** 128
 - **Max. Sequence Length:** 4096
 - **Loss Accumulation:** Sum (see https://unsloth.ai/blog/gradient)
 - **Learning Rate Schedule:** Linear

 ## Hyperparamters
 SFT:
+- **Learning Rate**: 5E-6 (8B), 2E-6 (70B, 405B)
+- **Effective Batch Size:** 128 (8B, 70B), 256 (405B)
 - **Max. Sequence Length:** 4096
 - **Loss Accumulation:** Sum (see https://unsloth.ai/blog/gradient)
 - **Learning Rate Schedule:** Linear