|
--- |
|
license: bsd |
|
--- |
|
|
|
Converted INT8/INT4 files for [fastllm](https://github.com/ztxz16/fastllm) with [baichuan2-13b-chat](https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat) |
|
|
|
Directly download from Baidu Netdisk: |
|
|
|
Link：https://pan.baidu.com/s/1Xsiif_1VzDyWFei1u5oJcA |
|
Code：wxbo |
|
|
|
Updated time: 2023/09/11 |
|
|
|
|
|
``` |
|
|
|
baichuan2-13b-chat-int8.flm: |
|
|
|
|===============================+======================+======================| |
|
| 0 NVIDIA GeForce ... Off | 00000000:05:00.0 Off | N/A | |
|
| 31% 36C P8 28W / 250W | 15420MiB / 22528MiB | 0% Default | |
|
+-------------------------------+----------------------+----------------------+ |
|
|
|
``` |
|
|
|
```python |
|
from fastllm_pytools import llm |
|
model = llm.model("baichuan2-13b-chat-int8.flm") |
|
for response in model.stream_response("介绍一下南京"): |
|
print(response, flush = True, end = "") |
|
``` |
|
> Note: please use the lastest version of FastLLM (no later than 2023/09/11 main branch) |
|
|
