metadata
license: other
license_name: govtech-singapore
license_link: LICENSE
Off-Topic Classification Model
Off-Topic Classification Model
This model leverages a fine-tuned Cross Encoder STSB Roberta Base to perform binary classification, determining whether a user prompt is off-topic in relation to the system's intended purpose as defined by the system prompt.
Model Highlights
- Base Model:
stsb-roberta-base
- Maximum Context Length: 1024 tokens
- Task: Binary classification (on-topic/off-topic)
Performance
Approach | Model | ROC-AUC | F1 | Precision | Recall |
---|---|---|---|---|---|
Fine-tuned cross-encoder classifier | stsb-roberta-base | 0.99 | 0.99 | 0.99 | 0.99 |
Pre-trained cross-encoder | stsb-roberta-base | 0.73 | 0.68 | 0.53 | 0.93 |
Usage
Clone this repository and install the required dependencies:
pip install -r requirements.txt
You can run the model using two options:
Option 1: Using
inference_onnx.py
with the ONNX Model.``` python inference_onnx.py '[ ["System prompt example 1", "User prompt example 1"], ["System prompt example 2", "System prompt example 2] ]' ```
Option 2: Using
inference_safetensors.py
with PyTorch and SafeTensors.``` python inference_safetensors.py '[ ["System prompt example 1", "User prompt example 1"], ["System prompt example 2", "System prompt example 2] ]' ```
Read more about this model in our technical report.