<img src="https://public.3.basecamp.com/p/rs5XqmAuF1iEuW6U7nMHcZeY/upload/download/VL-NLP-short.png" alt="logo voicelab nlp" style="width:300px;"/>

# SHerbert large - Polish SentenceBERT
SentenceBERT is a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. Training was based on the original paper [Siamese BERT models for the task of semantic textual similarity (STS)](https://arxiv.org/abs/1908.10084) with a slight modification of how the training data was used. The goal of the model is to generate different embeddings based on the semantic and topic similarity of the given text.

> Semantic textual similarity analyzes how similar two pieces of texts are.

Read more about how the model was prepared in our [blog post](https://voicelab.ai/blog/).

The base trained model is a Polish HerBERT. HerBERT is a BERT-based Language Model. For more details, please refer to: "HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish".

# Corpus
Te model was trained solely on [Wikipedia](https://dumps.wikimedia.org/).


# Tokenizer

As in the original HerBERT implementation, the training dataset was tokenized into subwords using a character level byte-pair encoding (CharBPETokenizer) with a vocabulary size of 50k tokens. The tokenizer itself was trained with a tokenizers library. 

We kindly encourage you to use the Fast version of the tokenizer, namely HerbertTokenizerFast.

# Usage

 ```python
from transformers import AutoTokenizer, AutoModel
from sklearn.metrics import pairwise

sbert = AutoModel.from_pretrained("Voicelab/sbert-large-cased-pl")
tokenizer = AutoTokenizer.from_pretrained("Voicelab/sbert-large-cased-pl")

s0 = "Uczenie maszynowe jest konsekwencją rozwoju idei sztucznej inteligencji i metod jej wdrażania praktycznego."
s1 = "Głębokie uczenie maszynowe jest sktukiem wdrażania praktycznego metod sztucznej inteligencji oraz jej rozwoju."
s2 = "Kasparow zarzucił firmie IBM oszustwo, kiedy odmówiła mu dostępu do historii wcześniejszych gier Deep Blue. "

tokens = tokenizer([s0, s1, s2], 
                    padding=True, 
                    truncation=True,
                    return_tensors='pt')
x = sbert(tokens["input_ids"],
            tokens["attention_mask"]).pooler_output

# similarity between sentences s0 and s1
print(pairwise.cosine_similarity(x[0], x[1])) # Result: 0.8011128

# similarity between sentences s0 and s2
print(pairwise.cosine_similarity(x[0], x[2])) # Result: 0.58822715
    
 ```
# Results

| Model                    | Accuracy   | Source                                                   |
|--------------------------|------------|----------------------------------------------------------|
| SBERT-WikiSec-base (EN)  | 80.42%     | https://arxiv.org/abs/1908.10084                         |
| SBERT-WikiSec-large (EN) | 80.78%     | https://arxiv.org/abs/1908.10084                         |
| sbert-base-cased-pl       | 82.31%     | https://huggingface.co/Voicelab/sbert-base-cased-pl    |
| **sbert-large-cased-pl**  | **84.42%** | **https://huggingface.co/Voicelab/sbert-large-cased-pl** |

# License

CC BY 4.0

# Citation

If you use this model, please cite the following paper:


# Authors

The model was trained by NLP Research Team at Voicelab.ai.

You can contact us [here](https://voicelab.ai/contact/).
{"language": ["pl"], "license": "cc-by-4.0", "tags": ["sentence-transformers", "feature-extraction", "sentence-similarity"], "datasets": ["Wikipedia"], "pipeline_tag": "sentence-similarity", "widget": [{"source_sentence": "Uczenie maszynowe jest konsekwencj\u0105 rozwoju idei sztucznej inteligencji i metod jej wdra\u017cania praktycznego.", "sentences": ["G\u0142\u0119bokie uczenie maszynowe jest sktukiem wdra\u017cania praktycznego metod sztucznej inteligencji oraz jej rozwoju.", "Kasparow zarzuci\u0142 firmie IBM oszustwo, kiedy odm\u00f3wi\u0142a mu dost\u0119pu do historii wcze\u015bniejszych gier Deep Blue. ", "Samica o d\u0142ugo\u015bci cia\u0142a 10\u201311 mm, szczoteczki na tylnych nogach s\u0142u\u017c\u0105ce do zbierania py\u0142ku oraz w\u0142oski na ko\u0144cu odw\u0142oka jaskrawo pomara\u0144czowoczerwone. "], "example_title": "Uczenie maszynowe"}]}
sentence-transformers pytorch bert feature-extraction sentence-similarity pl dataset:Wikipedia arxiv:1908.10084 license:cc-by-4.0 endpoints_compatible region:us
TAGS
#sentence-transformers #pytorch #bert #feature-extraction #sentence-similarity #pl #dataset-Wikipedia #arxiv-1908.10084 #license-cc-by-4.0 #endpoints_compatible #region-us 

<img src="https://public.3.URL alt="logo voicelab nlp" style="width:300px;"/>


SHerbert large - Polish SentenceBERT
====================================


SentenceBERT is a modification of the pretrained BERT network that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. Training was based on the original paper Siamese BERT models for the task of semantic textual similarity (STS) with a slight modification of how the training data was used. The goal of the model is to generate different embeddings based on the semantic and topic similarity of the given text.



> 
> Semantic textual similarity analyzes how similar two pieces of texts are.
> 
> 
> 


Read more about how the model was prepared in our blog post.


The base trained model is a Polish HerBERT. HerBERT is a BERT-based Language Model. For more details, please refer to: "HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish".


Corpus
======


Te model was trained solely on Wikipedia.


Tokenizer
=========


As in the original HerBERT implementation, the training dataset was tokenized into subwords using a character level byte-pair encoding (CharBPETokenizer) with a vocabulary size of 50k tokens. The tokenizer itself was trained with a tokenizers library.


We kindly encourage you to use the Fast version of the tokenizer, namely HerbertTokenizerFast.


Usage
=====


Results
=======


Model: SBERT-WikiSec-base (EN), Accuracy: 80.42%, Source: URL
Model: SBERT-WikiSec-large (EN), Accuracy: 80.78%, Source: URL
Model: sbert-base-cased-pl, Accuracy: 82.31%, Source: URL
Model: sbert-large-cased-pl, Accuracy: 84.42%, Source: URL


License
=======


CC BY 4.0


If you use this model, please cite the following paper:


Authors
=======


The model was trained by NLP Research Team at URL.


You can contact us here.



TAGS
#sentence-transformers #pytorch #bert #feature-extraction #sentence-similarity #pl #dataset-Wikipedia #arxiv-1908.10084 #license-cc-by-4.0 #endpoints_compatible #region-us 

passage: TAGS
#sentence-transformers #pytorch #bert #feature-extraction #sentence-similarity #pl #dataset-Wikipedia #arxiv-1908.10084 #license-cc-by-4.0 #endpoints_compatible #region-us 
