CIRCL/vulnerability-severity-classification-roberta-base

VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification

Severity classification

This model is a fine-tuned version of roberta-base on the dataset CIRCL/vulnerability-scores.

The model was presented in the paper VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification [arXiv].

Abstract: VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.

You can read this page for more information.

Model description

It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.

It achieves the following results on the evaluation set:

• Loss: 0.5098
• Accuracy: 0.8258

How to get started with the model

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

labels = ["low", "medium", "high", "critical"]

model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)

# Run inference
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

# Print results
print("Predictions:", predictions)
predicted_class = torch.argmax(predictions, dim=-1).item()
print("Predicted severity:", labels[predicted_class])

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.6421	1.0	28117	0.6400	0.7436
0.5734	2.0	56234	0.5903	0.7758
0.4304	3.0	84351	0.5422	0.7951
0.4694	4.0	112468	0.5055	0.8176
0.3141	5.0	140585	0.5098	0.8258

Framework versions

• Transformers 4.51.3
• Pytorch 2.7.1+cu126
• Datasets 3.6.0
• Tokenizers 0.21.1