A bot of this type would need to operate via more than just one invocation of the model. The process would go something like:
* A small speech classifier model, which is continually sampling small overlapping windows for an activation sequence. This would only require one output node, a flag indicating whether the model was activated.
* A larger speech to text model, which samples a much larger window that begins with the small window of the previous classifier, and contains enough of the following data to contain the requests. This would convert the request into text that could be passed into a traditional large language model such as Open Assistant.
* A Large Language model as mentioned above. This would be fine tuned to take the request and generate a formatted set of analyses like "Is this asking for a discord operation? If so, what operation? Is this asking for a web query? If so, what query? Is this asking for an LLM generated response? Does this require another processing step on the result of the query or LLM response?" in a simple JSON format. For example, it might return { "discord_action": "!add example_user voice_chat_channel" } for the input "hey voicebot, please add the user "example_user" to the voice chat channel.
* A fuzzy string matcher that matches returned arguments to commands to the relevant options. This is necessary as the speech to text may be providing a text output that is not an exact match for the users and channels actually present in the server.
* A portion of the containing script that executes the commands. If it is simply discord text messages as commands, it can just return the command as text output.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence