Any plan/idea to support to inference using beam search?
#51
by
GoZion
- opened
Hello! I try to inference using beam search (num_beams > 1), but the model does not support that. Do you have any plan to support to inference using beam search? Or how should I modify the modeling code to support that? Looking forward to your reply!
Hello @Gone! Yes, we definitely have plans to support beam search.
We just added support for training/fine-tuning with attention mask and GQA/MQA. The next step is to add support for Flash-Attention and finally generation with beam search!
Hello @Gone! Yes, we definitely have plans to support beam search.
We just added support for training/fine-tuning with attention mask and GQA/MQA. The next step is to add support for Flash-Attention and finally generation with beam search!
GoZion
changed discussion status to
closed
Thanks!