llama2-7-dolly-query

This model is a fine-tuned version of Llama-2-7b-chat-hf on the generator dataset. Can be used in conjunction with LukeOLuck/llama2-7-dolly-answer

Model description

A Fine-Tuned PEFT Adapter for the llama2 7b chat hf model Leverages FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, QLoRA: Efficient Finetuning of Quantized LLMs, and PEFT

Intended uses & limitations

Generate a query based on context and input

Training and evaluation data

Used SFTTrainer, checkout the code

Training procedure

Checkout the code here

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 32
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 64
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.03
  • num_epochs: 3

Training results

image/png

Framework versions

  • PEFT 0.8.2
  • Transformers 4.37.2
  • Pytorch 2.1.0+cu121
  • Datasets 2.17.1
  • Tokenizers 0.15.2
Downloads last month
18
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no pipeline_tag.

Model tree for LukeOLuck/llama2-7-dolly-query

Adapter
(2)
this model

Dataset used to train LukeOLuck/llama2-7-dolly-query