Can you officially support VLLM?
#48
by
SongXiaoMao
- opened
The model can be started normally with the latest version of VLLM, but the model is answered with a lot of exclamation points. But models quantified using AWQ don't have this problem.
Orion-zhen/QwQ-32B-Preview-AWQ
I've redownloaded the HF model and updated vllm to the latest version, 0.6.5, and it's working perfectly now. Thank you!
SongXiaoMao
changed discussion status to
closed