File size: 2,176 Bytes
cc5aa64 7aa2af3 6527e25 7aa2af3 cc5aa64 7aa2af3 f30968e 7aa2af3 f30968e 7aa2af3 6527e25 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
---
license: agpl-3.0
library_name: peft
tags:
- not-for-all-audiences
---
# Pippa-13b-qlora
This is a repository of my Llama-2-13b Qlora checkpoints of the [PIPPA-13b-ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-ShareGPT) dataset.
You can read more about the dataset on its relevant page. It's a ShareGPT reformat of the [PIPPA dataset](https://huggingface.co/datasets/PygmalionAI/PIPPA) by PygmalionAI. The reformat was done to allow for axolotl compatability.
### Architecture
- **Model Architecture**: Llama-2-13b
- **Training Algorithm**: QLora
- **Dataset Used**: PIPPA-ShareGPT (pippa_sharegpt_trimmed.jsonl)
### Training Details
- **Dataset**: [PIPPA-ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-ShareGPT)
- **Datset type**: ShareGPT
- **Training Parameters**: [See Here](https://gist.github.com/bdashore3/55ae04892f31609f2c3779c4a8a55408)
- **Training Environment**: Axolotl
- **sequence_len**: 4096
## Instruct Format
ShareGPT gets converted to vicuna format. The dataset uses modified roles of `USER` and `CHARACTER` instead of `USER` and `ASSISTANT`.
```
SYSTEM: Enter roleplay mode...
USER: {prompt}
CHARACTER:
```
## Notes
This Qlora was produced as an experiment to see how the public version of PIPPA can affect a model. As a result, I have no idea if this lora is of great quality or absolute garbage.
## Acknowledgments
Thanks to:
- PygmalionAI: The creators of the PIPPA dataset
- Axolotl: Finetuning suite
## Donate?
All my infrastructure and cloud expenses are paid out of pocket. If you'd like to donate, you can do so here: [https://ko-fi.com/kingbri](https://ko-fi.com/kingbri)
You should not feel obligated to donate, but if you do, I'd appreciate it.
## Axolotl stuff
## Training procedure
The following `bitsandbytes` quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
### Framework versions
- PEFT 0.6.0.dev0
|