

# SVALabs - Gbert Large Zeroshot Nli  

In this repository, we present our German zeroshot classification model. 

This model was trained on the basis of the German BERT large model from [deepset.ai](https://huggingface.co/deepset/gbert-large) and finetuned for natural language inference based on 847.862 machine-translated nli sentence pairs, using the [mnli](https://huggingface.co/datasets/multi_nli), [anli](https://huggingface.co/datasets/anli) and [snli](https://huggingface.co/datasets/snli) datasets. For this purpose, we translated the sentence pairs in these datasets to German.

If you are a German speaker you may also have a look at our [Blog post](https://focus.sva.de/zeroshot-klassifikation/) about this model and about Zeroshot Classification. 

### Model Details

| | Description or Link  |
|---|---|
|**Base model**   | [```gbert-large```](https://huggingface.co/deepset/gbert-large) |
|**Finetuning task**| Text Pair Classification / Natural Language Inference  |
|**Source datasets**| [```mnli```](https://huggingface.co/datasets/multi_nli); [```anli```](https://huggingface.co/datasets/anli); [```snli```](https://huggingface.co/datasets/snli)   |

### Performance

We evaluated our model for the nli task using the TEST set of the German part of the [xnli](https://huggingface.co/datasets/xnli) dataset.

XNLI TEST-Set Accuracy: 85.6% 


### Zeroshot Text Classification Task Benchmark 

We further tested our model for a zeroshot text classification task using a part of the [10kGNAD Dataset](https://tblock.github.io/10kGNAD/).
Specifically, we used all articles that were labeled "Kultur", "Sport", "Web", "Wirtschaft" and "Wissenschaft". 

The next table shows the results as well as a comparison with other German language and multilanguage zeroshot options performing the same task: 

| Model               | Accuracy |
|:-------------------:|:------:|
| Svalabs/gbert-large-zeroshot-nli     | 0.81 |
| Sahajtomar/German_Zeroshot | 0.76 |
| Symanto/xlm-roberta-base-snli-mnli-anli-xnli | 0.16 |
| Deepset/gbert-base | 0.65 |


### How to use 

The simplest way to use the model is the huggingface transformers pipeline tool. 
Just initialize the pipeline specifying the task as "zero-shot-classification" 
and select "svalabs/gbert-large-zeroshot-nli" as model. 

The model requires you to specify labels, 
a sequence (or list of sequences) to classify and a hypothesis template. 
In our tests, if the labels comprise only single words, 
"In diesem Satz geht es um das Thema {}" performed the best. 

However, for multiple words, especially when they combine nouns and verbs,
simple hypothesis such as "Weil {}" or "Daher {}" may work better. 

Here is an example of how to use the model:  

```python

from transformers import pipeline

zershot_pipeline = pipeline("zero-shot-classification",
                             model="svalabs/gbert-large-zeroshot-nli")

sequence = "Ich habe ein Problem mit meinem Iphone das so schnell wie möglich gelöst werden muss" 
labels = ["Computer", "Handy", "Tablet", "dringend", "nicht dringend"] 
hypothesis_template = "In diesem Satz geht es um das Thema {}."    


zershot_pipeline(sequence, labels, hypothesis_template=hypothesis_template)

```

### Contact
- Daniel Ehnes, daniel.ehnes@sva.de
- Baran Avinc, baran.avinc@sva.de

{"language": "German", "tags": ["text-classification", "pytorch", "nli", "de"], "pipeline_tag": "zero-shot-classification", "widget": [{"text": "Ich habe ein Problem mit meinem Iphone das so schnell wie m\u00f6glich gel\u00f6st werden muss.", "candidate_labels": "Computer, Handy, Tablet, dringend, nicht dringend", "hypothesis_template": "In diesem Satz geht es um das Thema {}."}]}
transformers pytorch bert text-classification nli de zero-shot-classification autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #bert #text-classification #nli #de #zero-shot-classification #autotrain_compatible #endpoints_compatible #region-us 

SVALabs - Gbert Large Zeroshot Nli
==================================


In this repository, we present our German zeroshot classification model.


This model was trained on the basis of the German BERT large model from URL and finetuned for natural language inference based on 847.862 machine-translated nli sentence pairs, using the mnli, anli and snli datasets. For this purpose, we translated the sentence pairs in these datasets to German.


If you are a German speaker you may also have a look at our Blog post about this model and about Zeroshot Classification.


### Model Details



### Performance


We evaluated our model for the nli task using the TEST set of the German part of the xnli dataset.


XNLI TEST-Set Accuracy: 85.6%


### Zeroshot Text Classification Task Benchmark


We further tested our model for a zeroshot text classification task using a part of the 10kGNAD Dataset.
Specifically, we used all articles that were labeled "Kultur", "Sport", "Web", "Wirtschaft" and "Wissenschaft".


The next table shows the results as well as a comparison with other German language and multilanguage zeroshot options performing the same task:



### How to use


The simplest way to use the model is the huggingface transformers pipeline tool.
Just initialize the pipeline specifying the task as "zero-shot-classification"
and select "svalabs/gbert-large-zeroshot-nli" as model.


The model requires you to specify labels,
a sequence (or list of sequences) to classify and a hypothesis template.
In our tests, if the labels comprise only single words,
"In diesem Satz geht es um das Thema {}" performed the best.


However, for multiple words, especially when they combine nouns and verbs,
simple hypothesis such as "Weil {}" or "Daher {}" may work better.


Here is an example of how to use the model:


### Contact


* Daniel Ehnes, URL@URL
* Baran Avinc, URL@URL



### Model Details ### Performance


We evaluated our model for the nli task using the TEST set of the German part of the xnli dataset.


XNLI TEST-Set Accuracy: 85.6% ### Zeroshot Text Classification Task Benchmark


We further tested our model for a zeroshot text classification task using a part of the 10kGNAD Dataset.
Specifically, we used all articles that were labeled "Kultur", "Sport", "Web", "Wirtschaft" and "Wissenschaft".


The next table shows the results as well as a comparison with other German language and multilanguage zeroshot options performing the same task: ### How to use


The simplest way to use the model is the huggingface transformers pipeline tool.
Just initialize the pipeline specifying the task as "zero-shot-classification"
and select "svalabs/gbert-large-zeroshot-nli" as model.


The model requires you to specify labels,
a sequence (or list of sequences) to classify and a hypothesis template.
In our tests, if the labels comprise only single words,
"In diesem Satz geht es um das Thema {}" performed the best.


However, for multiple words, especially when they combine nouns and verbs,
simple hypothesis such as "Weil {}" or "Daher {}" may work better.


Here is an example of how to use the model: ### Contact


* Daniel Ehnes, URL@URL
* Baran Avinc, URL@URL
TAGS
#transformers #pytorch #bert #text-classification #nli #de #zero-shot-classification #autotrain_compatible #endpoints_compatible #region-us 
 ### Model Details ### Performance


We evaluated our model for the nli task using the TEST set of the German part of the xnli dataset.


XNLI TEST-Set Accuracy: 85.6% ### Zeroshot Text Classification Task Benchmark


We further tested our model for a zeroshot text classification task using a part of the 10kGNAD Dataset.
Specifically, we used all articles that were labeled "Kultur", "Sport", "Web", "Wirtschaft" and "Wissenschaft".


The next table shows the results as well as a comparison with other German language and multilanguage zeroshot options performing the same task: ### How to use


The simplest way to use the model is the huggingface transformers pipeline tool.
Just initialize the pipeline specifying the task as "zero-shot-classification"
and select "svalabs/gbert-large-zeroshot-nli" as model.


The model requires you to specify labels,
a sequence (or list of sequences) to classify and a hypothesis template.
In our tests, if the labels comprise only single words,
"In diesem Satz geht es um das Thema {}" performed the best.


However, for multiple words, especially when they combine nouns and verbs,
simple hypothesis such as "Weil {}" or "Daher {}" may work better.


Here is an example of how to use the model: ### Contact


* Daniel Ehnes, URL@URL
* Baran Avinc, URL@URL
passage: TAGS
#transformers #pytorch #bert #text-classification #nli #de #zero-shot-classification #autotrain_compatible #endpoints_compatible #region-us 
### Model Details### Performance


We evaluated our model for the nli task using the TEST set of the German part of the xnli dataset.


XNLI TEST-Set Accuracy: 85.6%### Zeroshot Text Classification Task Benchmark


We further tested our model for a zeroshot text classification task using a part of the 10kGNAD Dataset.
Specifically, we used all articles that were labeled "Kultur", "Sport", "Web", "Wirtschaft" and "Wissenschaft".


The next table shows the results as well as a comparison with other German language and multilanguage zeroshot options performing the same task:### How to use


The simplest way to use the model is the huggingface transformers pipeline tool.
Just initialize the pipeline specifying the task as "zero-shot-classification"
and select "svalabs/gbert-large-zeroshot-nli" as model.


The model requires you to specify labels,
a sequence (or list of sequences) to classify and a hypothesis template.
In our tests, if the labels comprise only single words,
"In diesem Satz geht es um das Thema {}" performed the best.


However, for multiple words, especially when they combine nouns and verbs,
simple hypothesis such as "Weil {}" or "Daher {}" may work better.


Here is an example of how to use the model:### Contact


* Daniel Ehnes, URL@URL
* Baran Avinc, URL@URL