license: other
Koala: A Dialogue Model for Academic Research
This repo contains the weights of the Koala 13B model produced at Berkeley. It is the result of combining the diffs from https://huggingface.co/young-geng/koala with the original Llama 13B model.
This version has then been quantized to 4bit using https://github.com/qwopqwop200/GPTQ-for-LLaMa
Other Koala repos
These other versions are also available:
- Unquantized 13B model in HF format
- Unquantized 7B model in HF format
- Unquantized 7B model in GGML format for llama.cpp
Quantization method
This GPTQ model was quantized using GPTQ-for-LLaMa with the following command:
python3 llama.py /content/koala-13B-HF c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save /content/koala-13B-4bit-128g.pt
How to run with text-generation-webui
The model files provided will not load as-is with oobaboogas text-generation-webui.
They require the latest version of the GPTQ code.
Here are the commands I used to clone GPTQ-for-LLaMa, clone text-generation-webui, and install GPTQ into the UI:
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa
git clone https://github.com/oobabooga/text-generation-webui
mkdir -p text-generation-webui/repositories
ln -s GPTQ-for-LLaMa text-generation-webui/repositories/GPTQ-for-LLaMa
Then install this model into text-generation-webui/models and run text-generation-webui as follows:
cd text-generation-webui
python server.py --model koala-13B-4bit-128g --wbits 4 --groupsize 128 --model_type Llama
Coming soon
Tomorrow I will upload a safetensors file as well.
How to merge Koala delta weights
The Koala delta weights were originally merged using the following commands, producing koala-13B-HF:
git clone https://github.com/young-geng/EasyLM
git clone https://huggingface.co/TheBloke/llama-13b
mkdir koala_diffs && cd koala_diffs && wget https://huggingface.co/young-geng/koala/resolve/main/koala_13b_diff_v2
cd EasyLM
PYTHON_PATH="${PWD}:$PYTHONPATH" python \
-m EasyLM.models.llama.convert_torch_to_easylm \
--checkpoint_dir=/content/llama-13b \
--output_file=/content/llama-13b-LM \
--streaming=True
PYTHON_PATH="${PWD}:$PYTHONPATH" python \
-m EasyLM.scripts.diff_checkpoint --recover_diff=True \
--load_base_checkpoint='params::/content/llama-13b-LM' \
--load_target_checkpoint='params::/content/koala_diffs/koala_13b_diff_v2' \
--output_file=/content/koala_13b.diff.weights \
--streaming=True
PYTHON_PATH="${PWD}:$PYTHONPATH" python \
-m EasyLM.models.llama.convert_easylm_to_hf --model_size=7b \
--output_dir=/content/koala-13B-HF \
--load_checkpoint='params::/content/koala_13b.diff.weights' \
--tokenizer_path=/content/llama-13b/tokenizer.model
Check out the following links to learn more about the Berkeley Koala model.
- Blog post
- Online demo
- EasyLM: training and serving framework on GitHub
- Documentation for running Koala locally
License
The model weights are intended for academic research only, subject to the model License of LLaMA, Terms of Use of the data generated by OpenAI, and Privacy Practices of ShareGPT. Any other usage of the model weights, including but not limited to commercial usage, is strictly prohibited.