|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- distilbert/distilbert-base-multilingual-cased |
|
|
pipeline_tag: text-classification |
|
|
library_name: transformers |
|
|
tags: |
|
|
- code |
|
|
- cyber |
|
|
--- |
|
|
|
|
|
|
|
|
# Transformer |
|
|
|
|
|
This is a transformers model fine tuned for malicious URL detection. Given a FQDN URL it outputs probability of it to be malicious identifying common suspicious pattern. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
- **Developed by:** Anvilogic |
|
|
- **Model Type:** Transformer |
|
|
- **Maximum Sequence Length:** 512 tokens |
|
|
- **Output Dimensionality:** 768 tokens |
|
|
- **Finetuned from model:** [distilbert](https://huggingface.co/distilbert/distilbert-base-cased) |
|
|
- **Language(s) (NLP):** Multilingual |
|
|
- **License:** MIT |
|
|
|
|
|
### Full Model Architecture |
|
|
|
|
|
``` |
|
|
DistilBERT: |
|
|
name: "distilbert-base-cased" |
|
|
params: |
|
|
layers: 6 |
|
|
hidden_size: 768 |
|
|
attention_heads: 12 |
|
|
ff_dim: 3072 |
|
|
max_seq_len: 512 |
|
|
vocab_size: 28996 |
|
|
total_params: 66M |
|
|
activation: "gelu" |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Direct Usage |
|
|
|
|
|
First install the Transformers library: |
|
|
|
|
|
```bash |
|
|
pip install -U transformers |
|
|
``` |
|
|
Then you can load this model and run inference. |
|
|
```python |
|
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
import torch |
|
|
# Load pre-trained model and tokenizer |
|
|
model_name = "Anvilogic/URLGuardian" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2) # Adjust `num_labels` based on your task |
|
|
# Example sentences |
|
|
sentences = ["paypal.com.secure-login.xyz","bit.ly/fake-login"] |
|
|
# Tokenize inputs |
|
|
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt") |
|
|
# Run inference |
|
|
with torch.no_grad(): |
|
|
outputs = model(**inputs) |
|
|
logits = outputs.logits # Raw predictions |
|
|
predictions = torch.argmax(logits, dim=-1) # Convert to class labels |
|
|
# Print results |
|
|
print(predictions.tolist()) # Example output: [1, 0] (Assuming 1 = Positive, 0 = Negative) |
|
|
``` |
|
|
### Downstream Usage |
|
|
This dataset enables real-time malicious URL detection with lightweight models, supporting large-scale inference for phishing prevention and cybersecurity monitoring. |
|
|
## Training Details |
|
|
|
|
|
### Framework Versions |
|
|
- Python: 3.10.14 |
|
|
- Transformers: 4.49.0 |
|
|
- PyTorch: 2.2.2 |
|
|
- Tokenizers: 0.20.3 |
|
|
|
|
|
### Training Data |
|
|
|
|
|
The model was fine-tuned using [Anvilogic/URL-Guardian-Dataset](https://huggingface.co/datasets/Anvilogic/URL-Guardian-Dataset), which contains URL as well as their labels. |
|
|
The dataset was filtered and converted to the parquet format for efficient processing. |
|
|
|
|
|
### Training Procedure |
|
|
The model was optimized using [BCELoss](https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html) |
|
|
|
|
|
#### Training Hyperparameters |
|
|
- **Model Architecture**: encoder fine-tuned from [distilbert](https://huggingface.co/distilbert/distilbert-base-cased) |
|
|
- **Batch Size**: 32 |
|
|
- **Epochs**: 3 |
|
|
- **Learning Rate**: 2e-5 |
|
|
- **Warmup Steps**: 100 |
|
|
|
|
|
|
|
|
## Evaluation |
|
|
|
|
|
In the final evaluation after training, the model achieved the following metrics on the test set: |
|
|
|
|
|
**Binary Classification Evaluator** |
|
|
```json |
|
|
Accuracy : 0.9744 |
|
|
F1 Score : 0.9742 |
|
|
Precision : 0.9771 |
|
|
Recall : 0.9712 |
|
|
Average Precision : 0.9962 |
|
|
``` |
|
|
These results indicate the model's high performance in identifying maliciosu URLs, with strong precision and recall scores that make it well-suited for cybersecurity applications. |
