|
|
--- |
|
|
license: eupl-1.2 |
|
|
language: code |
|
|
base_model: |
|
|
- NetherlandsForensicInstitute/ARM64BERT |
|
|
library_name: sentence-transformers |
|
|
--- |
|
|
|
|
|
ARM64BERT-embedding 🦾 |
|
|
====================== |
|
|
|
|
|
[GitHub repository](https://github.com/NetherlandsForensicInstitute/asmtransformers) |
|
|
|
|
|
## General |
|
|
### What is the purpose of the model |
|
|
The model is a BERT model of ARM64 assembly code that can be used to find similar ARM64 functions to a given ARM64 function. |
|
|
This task is known as _binary code similarity detection_, which is similar to the _sentence similarity_ task in natural language processing. |
|
|
|
|
|
### What does the model architecture look like? |
|
|
The model architecture is inspired by [jTrans](https://github.com/vul337/jTrans) (Wang et al., 2022). |
|
|
It is a BERT model (Devlin et al. 2019) although the typical Next Sentence Prediction has been replaced with Jump Target Prediction, as proposed in Wang et al. |
|
|
This architecture has subsequently been finetuned for semantic search purposes. We have followed the procedure proposed by [S-BERT](https://www.sbert.net/examples/applications/semantic-search/README.html). |
|
|
|
|
|
### What is the output of the model? |
|
|
The model returns an embedding vector of 768 dimensions for each function that it's given. These embeddings can be compared to |
|
|
get an indication of which functions are similar to each other. |
|
|
|
|
|
### How does the model perform? |
|
|
The model has been evaluated on [Mean Reciprocal Rank (MRR)](https://en.wikipedia.org/wiki/Mean_reciprocal_rank) and |
|
|
[Recall@1](https://en.wikipedia.org/wiki/Precision_and_recall). |
|
|
When the model has to pick the positive example out of a pool of 32, ranks the positive example highest most of the time. |
|
|
When the pool is significantly enlarged to 10.000 functions, it still ranks the positive example first or second in most cases. |
|
|
|
|
|
| Model | Pool size | MRR | Recall@1 | |
|
|
|----------------------|-----------|------|----------| |
|
|
| ARM64BERT | 32 | 0.78 | 0.72 | |
|
|
| ARM64BERT-embedding | 32 | 0.99 | 0.99 | |
|
|
| ARM64BERT | 10.000 | 0.58 | 0.56 | |
|
|
| ARM64BERT-embedding | 10.000 | 0.87 | 0.83 | |
|
|
|
|
|
## Purpose and use of the model |
|
|
|
|
|
### For which problem has the model been designed? |
|
|
The model has been designed to find similar ARM64 functions in a database of known ARM64 functions. |
|
|
|
|
|
### What else could the model be used for? |
|
|
We do not see other applications for this model. |
|
|
|
|
|
### To what problems is the model not applicable? |
|
|
This model has been finetuned on the semantic search task. |
|
|
For the base ARM64BERT model, please refer to the [other |
|
|
model](https://huggingface.co/NetherlandsForensicInstitute/ARM64BERT) we have published. |
|
|
|
|
|
## Data |
|
|
### What data was used for training and evaluation? |
|
|
The dataset is created in the same way as Wang et al. created Binary Corp. |
|
|
A large set of source code comes from the [ArchLinux official repositories](https://archlinux.org/packages/) and the [ArchLinux user repositories](https://aur.archlinux.org/packages/). |
|
|
All this code is split into functions that are compiled into binary code with different optimalizations |
|
|
(`O0`, `O1`, `O2`, `O3` and `Os`) and security settings (fortify or no-fortify). |
|
|
This results in a maximum of 10 (5×2) different functions which are semantically similar, i.e. they represent the same functionality, but have different machine code. |
|
|
The dataset is split into a train and a test set. This is done on project level, so all binaries and functions belonging to one project are part of |
|
|
either the train or the test set, not both. We have not performed any deduplication on the dataset for training. |
|
|
|
|
|
| set | # functions | |
|
|
|-------|------------:| |
|
|
| train | 18,083,285 | |
|
|
| test | 3,375,741 | |
|
|
|
|
|
For our training and evaluation code, see our [GitHub repository](https://github.com/NetherlandsForensicInstitute/asmtransformers). |
|
|
|
|
|
|
|
|
### By whom was the dataset collected and annotated? |
|
|
The dataset was collected by our team. |
|
|
|
|
|
### Any remarks on data quality and bias? |
|
|
After training our models, we found out that something had gone wrong when compiling our dataset. |
|
|
Consequently, the first line of the next function was included in the previous. |
|
|
This has been fixed for the finetuning, but due to the long training process, |
|
|
and the good performance of the model despite the mistake, we have decided not to retrain the base model. |
|
|
|
