README.md

metadata

language:
  - en

Text Classification of conversation flow

This a ONNX quantized model and is fined-tuned version of nreimers/MiniLMv2-L6-H384-distilled-from-RoBERTa-Large. The original model can be found here

A flow label is orthogonal to the main conversation goal, implying that it categorizes actions or responses in a way that is independent from the primary objective of the conversation.

This model should be used only for agent dialogs.

Usage

Installation

pip install tokenizers
pip install onnxruntime
git clone https://huggingface.co/minuva/MiniLMv2-agentflow-v2-onnx

Run the Model

import os
import numpy as np
import json

from tokenizers import Tokenizer
from onnxruntime import InferenceSession


model_name = "minuva/MiniLMv2-agentflow-v2-onnx"

tokenizer = Tokenizer.from_pretrained(model_name)
tokenizer.enable_padding(
    pad_token="<pad>",
    pad_id=1,
)
tokenizer.enable_truncation(max_length=256)
batch_size = 16

texts = ["thats my mistake"]
outputs = []
model = InferenceSession("MiniLMv2-agentflow-v2-onnx/model_optimized_quantized.onnx", providers=['CPUExecutionProvider'])

with open(os.path.join("MiniLMv2-agentflow-v2-onnx", "config.json"), "r") as f:
            config = json.load(f)

output_names = [output.name for output in model.get_outputs()]
input_names = [input.name for input in model.get_inputs()]

for subtexts in np.array_split(np.array(texts), len(texts) // batch_size + 1):
            encodings = tokenizer.encode_batch(list(subtexts))
            inputs = {
                "input_ids": np.vstack(
                    [encoding.ids for encoding in encodings],
                ),
                "attention_mask": np.vstack(
                    [encoding.attention_mask for encoding in encodings],
                ),
                "token_type_ids": np.vstack(
                    [encoding.type_ids for encoding in encodings],
                ),
            }

            for input_name in input_names:
                if input_name not in inputs:
                    raise ValueError(f"Input name {input_name} not found in inputs")

            inputs = {input_name: inputs[input_name] for input_name in input_names}
            output = np.squeeze(
                np.stack(
                    model.run(output_names=output_names, input_feed=inputs)
                ),
                axis=0,
            )
            outputs.append(output)

outputs = np.concatenate(outputs, axis=0)
scores = 1 / (1 + np.exp(-outputs))
results = []
for item in scores:
    labels = []
    scores = []
    for idx, s in enumerate(item):
        labels.append(config["id2label"][str(idx)])
        scores.append(float(s))
    results.append({"labels": labels, "scores": scores})


res = []

for result in results:
    joined = list(zip(result['labels'], result['scores']))
    max_score = max(joined, key=lambda x: x[1])    
    res.append(max_score)

res
# [('agent_apology_error_mistake', 0.9991708993911743)]

Categories Explanation

Click to expand!

- OTHER: Responses or actions by the agent that do not fit into the predefined categories or are outside the scope of the specific interactions listed.

- agent_apology_error_mistake: When the agent acknowledges an error or mistake in the information provided or in the handling of the request.

- agent_apology_unsatisfactory: The agent expresses an apology for providing an unsatisfactory response or for any dissatisfaction experienced by the user.

- agent_didnt_understand: Indicates that the agent did not understand the user's request or question.

- agent_limited_capabilities: The agent communicates its limitations in addressing certain requests or providing certain types of information.

- agent_refuses_answer: When the agent explicitly refuses to answer a question or fulfill a request, due to policy restrictions or ethical considerations.

- image_limitations": The agent points out limitations related to handling or interpreting images.

- no_information_doesnt_know": The agent indicates that it has no information available or does not know the answer to the user's question.

- success_and_followup_assistance": The agent successfully provides the requested information or service and offers further assistance or follow-up actions if needed.

Metrics in our private test dataset

Model (params)	Loss	Accuracy	F1
minuva/MiniLMv2-agentflow-v2 (33M)	0.1462	0.9773	0.9774
minuva/MiniLMv2-agentflow-v2-onnx (33M)	-	0.97394	0.97392

Deployment

Check our repository to see how to easily deploy this (quantized) model in a serverless environment with fast CPU inference and light resource utilization.