Could you please tell how to inference this model?

by carlosbdw - opened Dec 30, 2024

Dec 30, 2024

thank you very much , I used to use vllm ,but it doesn't work with it.

Dec 30, 2024

yes please

Jan 3

Any specific ideas on how to infer with vllm?

Jan 3

you can't. it's tensorRT

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment