language: en
library_name: LogClassifier
tags:
- log-classification
- log feature
- log-similarity
- transformers
- AIOps
pipeline_tag: text-classification
log-classifier-BERT-v1
log-classifier-v1 is a neural network-based log classification model, trained from BERTForSequenceClassification designed for use in network and device log mining tasks. Developed by Selector AI
Model Usage
from transformers import BertForSequenceClassification, BertTokenizer
# Step 1: Load the model and tokenizer from Hugging Face
model = BertForSequenceClassification.from_pretrained("rahulm-selector/log-classifier-BERT-v1")
tokenizer = BertTokenizer.from_pretrained("rahulm-selector/log-classifier-BERT-v1")
import torch
model.eval()
# Step 2: Prepare the input data (Example log text)
log_text = "Error occurred while accessing the database."
# Tokenize the input data
inputs = tokenizer(log_text, return_tensors="pt", padding=True, truncation=True, max_length=128)
# Step 3: Make predictions
with torch.no_grad():
outputs = model(**inputs)
logits = outputs.logits
# Step 4: Get the predicted class (the class with the highest score)
predicted_class = torch.argmax(logits, dim=1).item()
# label mapping (can load from JSON file in repo or config)
label_mapping = model.config.id2label
# Step 5: Get the event name
predicted_event = label_mapping[predicted_class]
print(f"Predicted Event: {predicted_event}")
Background
The model focuses on structured and semi-structured log data, outputing around 60 different event categories. It is highly effective for real-time log analysis, anomaly detection, and operational monitoring, helping organizations manage large-scale network data by automatically classifying logs into predefined categories, facilitating faster and more accurate diagnosis of network issues. The log-classifier-BERT-v1 model is designed to process logs as input and output a corresponding classification.
Intended uses
Our model is intended to be used as classifier. Given an input text (a log coming from a network/device), it outputs the corresponding event most associated with the log. The possible events that can be classified are shown in encoder.json
Training Details
Data
The model was trained on log data sourced from various network and infrastructure devices, capturing crucial system events and performance metrics. Syslogs originated from network routers, switches, firewalls, and servers, providing a rich dataset of operational insights including security events, traffic patterns, and hardware health statuses.
Train/Test Split
- Train Data Size:
~80K Logs - Test Data Size:
~20K Logs
Hyper Parameters
The following hyperparameters were used during training to optimize the model's performance:
- Batch Size:
32 - Learning Rate:
.001 - Optimizer:
Adam - Epochs:
10 - Dropout Rate: N/A
- LSTM Hidden Dimension:
384 - Embedding Dimension:
384