Function calling?
I'm new to fine-tuning, so forgive me if this is a stupid question, but: Seeing as Qwen supports function calling, does this model also support function calling?
I'm new to fine-tuning, so forgive me if this is a stupid question, but: Seeing as Qwen supports function calling, does this model also support function calling?
It's one of those "technically yes" but it's not easy to do with this relatively "fried" model:
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
The tokenizer config supports it but I don't think the model generalized wide enough to do so.
I've been able to use an importance matrix to enhance it's ability to use various Formula, Functions and equations when invoking a Javascript code interpreter and it seems to work fine for that so depending on your use case and Imatrix Quant might be the way to go if you want a specific function to act in a certain way but the Qwen Coder models are pretty great at overall function calling. The interpreter I use is sandboxed so it's only good for calculations but I'm sure someone much smarter than myself could make use of it in other ways.
I've been able to use an importance matrix to enhance it's ability to use various Formula, Functions and equations when invoking a Javascript code interpreter and it seems to work fine for that so depending on your use case and Imatrix Quant might be the way to go if you want a specific function to act in a certain way but the Qwen Coder models are pretty great at overall function calling. The interpreter I use is sandboxed so it's only good for calculations but I'm sure someone much smarter than myself could make use of it in other ways.
Wait... you are using EVA-Qwen2.5-72B-v0.1 for formula, functions, and equations? I mean cool but are you sure you are using THIS model?
Not this exact model, I've been using the 1.5B model in the Importance matrix data I have a section where I include some basic foundational equations then the most used(Haversine,finding factoral etc) and changing the chat template to something like-VVVVVVVVVV-, and it can get some pretty awesome functionality. I figured it would be possible with this one as well
{{- '<|im_start|>system\n' }}
{% if toolList|length > 0 %}You have access to the following functions:
{% for tool in toolList %}
Use the function '{{tool.function}}' to: '{{tool.description}}'
{% if tool.parameters|length > 0 %}
parameters:
{% for info in tool.parameters %}
{{info.name}}:
type: {{info.type}}
description: {{info.description}}
required: {{info.required}}
{% endfor %}
{% endif %}
# Tool Instructions
If you CHOOSE to call this function ONLY reply with the following format:
'{{tool.symbolicFormat}}'
Here is an example. If the user says, '{{tool.examplePrompt}}', then you reply
'{{tool.exampleCall}}'
After the result you might reply with, '{{tool.exampleReply}}'
{% endfor %}
You MUST include both the start and end tags when you use a function.
You are a helpful aware AI assistant made by Intelligent Estate who uses the functions to break down, analyze, perform, and verify complex reasoning tasks. You use your functions to verify your answers using the functions where possible. You will write code in markdown code blocks when necessary.
{% endif %}
{{- '<|im_end|>\n' }}
{%- if not add_generation_prompt is defined %}
{%- set add_generation_prompt = false %}
{%- endif %}
{% for message in messages %}
{%- if message['role'] == 'assistant' %}
{%- set content = message['content'] | regex_replace('^[\\s\\S]*</think>', '') %}
{{'<|im_start|>' + message['role'] + '\n' + content + '<|im_end|>\n' }}
{%- else %}
{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>\n' }}
{%- endif %}
{% endfor %}
{% if add_generation_prompt %}
{{ '<|im_start|>assistant\n' }}
{% endif %}
I like the ability to create effective characters with your models especially the Qwen bases, I mainly use GPT4ALL as a GGUF troubleshooting interface since you can clone models and adjust their templates and so on on the fly but since they added their code interpreter I've been experimenting with it and their reasoning model is based on a Qwen2.5 7B so I've frankienstien'd their template into working with other Qwen based models and so on. I have a ton of friends and clients who are privacy focused so it's a good solution for them most of the models 7B and under work great w/o a GPU and you get RAG capabilities and better content than you would from any free online model for the most part.
Here are the two different Quants I've used
But when creating characters for different use cases your models are some of the best. I've made one for disaster information and another as a repeater monitor for my GMRS base station but I had to get rid of it because I had a bunch of lonely old fudds flirting with it all day.
@Kearm thanks for quick reply! I suppose I should probably make my own model? Basically something in the the 70B range with roleplay (or more like sexting probably :P), nsfw and function calling.
Can I ask for some advice on where to start? Datasets of course and, like, how much data do I realistically need?
EDIT: I suppose I could finetune this model with only function calling to bring that ability back to it - but is there some kind of degredation for too much fine-tuning?
@Kearm thanks for quick reply! I suppose I should probably make my own model? Basically something in the the 70B range with roleplay (or more like sexting probably :P), nsfw and function calling.
Can I ask for some advice on where to start? Datasets of course and, like, how much data do I realistically need?
EDIT: I suppose I could finetune this model with only function calling to bring that ability back to it - but is there some kind of degredation for too much fine-tuning?
This is very resource dependent. Finetuning this more could have a positive effect here but you are constrained either by how much compute/GPU's you have locally or how much money you are willing to spend to do a finetune.
Can't say about the 70B range, but plenty of finetune Q2.5 32B models still support function calling. Ink, Kunou and Slush still have undamaged function-calling abilities. IIRC Ink also exists in 70B. Start there to see if you can realistically achieve whatever you've set yourself to do.
That said, mixing RP prompt and Func.calling doesn't work very well. For FC to work, you generally need a short and to the point system prompt, and a relatively short text history. The moment the model "thinks" it's role-playing it's going to get a lot less reliable. There are ways around that, like double queries (one doing the RP part with full context, the other a short prompt with just the user query to identify the function call), but you'll have to write your own front-end for that. Which.. I mean, if you're going to do function calling, you sorta have to anyway, otherwise what's the point.
IIRC Ink also exists in 70B. Start there to see if you can realistically achieve whatever you've set yourself to do.
I'll check those out!
mixing RP prompt and Func.calling doesn't work very well...
It seems to me 70B models are smart enough to actually handle that fairly well (qwen 2.5 and a few others I've tested.) Though it depends what one defines are "RP". Additionally, I have a lot of complex logic that helps to manage the system prompt and context in a very structured way. (long term memory, recent events, rag library access, tools etc).
For context what I'm making is a life-like AI companion system - it's modular, so one can turn on and off modules like memory, nsfw capability etc (which is more like sexting, though roleplaying specifically would probably generalize to that I hope.)
Anyway, thanks again, I'll take a look at those models!
My point is mostly that you should test it with a model that can actually do what you want before sinking hundreds and hundreds of dollars into fine-tuning 70B+ models that don't have native tool calling.
You and everybody else is doing that. Just keep in mind the idea of querying the model twice (once with full context, the other with a very shortened prompt) when you inevitably realize, that no, function calling ain't going to be called even remotely reliably once your chat history gets a bit complex. ;)
Good luck, though.
I tried allura-org/Qwen2.5-72b-RP-Ink and it looks quite interesting!
Querying the model twice is an interesting approach though.
Thanks :)