See axolotl config

axolotl version: 0.8.0.dev0

base_model: meta-llama/Llama-3.1-8B-Instruct
# optionally might have model_type or tokenizer_type
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer
# Automatically upload checkpoint and final model to HF
hub_model_id: seacorn/llama3.1-8b-reasoning-summarizer

load_in_8bit: true
load_in_4bit: false
strict: false

seed: 42

datasets:
  - path: output.jsonl
    type: chat_template
dataset_prepared_path:
val_set_size: 0.05
output_dir: ./lora-out

sequence_len: 8192
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_modules_to_save:
  - embed_tokens
  - lm_head

wandb_project: huggingface
wandb_entity:
wandb_watch:
wandb_name: llama3.1-8b-reasoning-summarizer
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 2
num_epochs: 2
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
s2_attention:

warmup_ratio: 0.05
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 5
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
   pad_token: <|end_of_text|>

llama3.1-8b-reasoning-summarizer

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the seacorn/news-summarizer-reasoner dataset. It achieves the following results on the evaluation set:

• Loss: 1.1173

Intended uses & limitations

The model performs best in summarization tasks, specifically in English and maybe Chinese. The model provides reasoning ON/OFF via system prompt trigger, all instructions should be contained within the user prompt.

Reasoning on example:

messages = [
    {"role": "system", "content": "reasoning on"},
    {"role": "user", "content": "Summarize the following into 5 bullet points, each with 20 words max.\n\nMarch 28 (Reuters) -..."}
]

# output
- Elon Musk's xAI acquires X ...

Reasoning off example:

messages = [
    {"role": "system", "content": "reasoning off"},
    {"role": "user", "content": "Summarize the following into 5 bullet points, each with 20 words max.\n\nMarch 28 (Reuters) -..."}
]

# output
<think>
Okay, I need to summarize this article into 5 bullet points, each with a maximum of 20 words. ...
</think>

- Musk's xAI acquires X ...

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 56
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.0396	0.0018	1	1.7982
1.3908	0.2506	141	1.2241
1.8534	0.5011	282	1.1842
1.5745	0.7517	423	1.1560
0.9261	1.0018	564	1.1288
1.2359	1.2523	705	1.1344
1.1835	1.5029	846	1.1223
0.9898	1.7534	987	1.1173

Framework versions

• PEFT 0.15.0
• Transformers 4.50.0
• Pytorch 2.5.1+cu124
• Datasets 3.4.1
• Tokenizers 0.21.1