# TinyBERT_L-4_H-312_v2 English Sentence Encoder

This is distilled from the `bert-base-nli-stsb-mean-tokens` pre-trained model from [Sentence-Transformers](https://sbert.net/).

The embedding vector is obtained by mean/average pooling of the last layer's hidden states.

Update 20210325: Added the attention matrices imitation objective as in the TinyBERT paper, and the distill target has been changed from `distilbert-base-nli-stsb-mean-tokens` to `bert-base-nli-stsb-mean-tokens` (they have almost the same STSb performance).

## Model Comparison

We compute cosine similarity scores of the embeddings of the sentence pair to get the spearman correlation on the STS benchmark (bigger is better):

|                                      | Dev   | Test  |
| ------------------------------------ | ----- | ----- |
| bert-base-nli-stsb-mean-tokens       | .8704 | .8505 |
| distilbert-base-nli-stsb-mean-tokens | .8667 | .8516 |
| TinyBERT_L-4_H-312_v2-distill-AllNLI | .8587 | .8283 |
| TinyBERT_L-4_H (20210325)            | .8551 | .8341 |

{}
transformers pytorch jax bert feature-extraction endpoints_compatible region:us
TAGS
#transformers #pytorch #jax #bert #feature-extraction #endpoints_compatible #region-us 

TinyBERT\_L-4\_H-312\_v2 English Sentence Encoder
=================================================


This is distilled from the 'bert-base-nli-stsb-mean-tokens' pre-trained model from Sentence-Transformers.


The embedding vector is obtained by mean/average pooling of the last layer's hidden states.


Update 20210325: Added the attention matrices imitation objective as in the TinyBERT paper, and the distill target has been changed from 'distilbert-base-nli-stsb-mean-tokens' to 'bert-base-nli-stsb-mean-tokens' (they have almost the same STSb performance).


Model Comparison
----------------


We compute cosine similarity scores of the embeddings of the sentence pair to get the spearman correlation on the STS benchmark (bigger is better):


Dev: bert-base-nli-stsb-mean-tokens, Test: .8704
Dev: distilbert-base-nli-stsb-mean-tokens, Test: .8667
Dev: TinyBERT\_L-4\_H-312\_v2-distill-AllNLI, Test: .8587
Dev: TinyBERT\_L-4\_H (20210325), Test: .8551



TAGS
#transformers #pytorch #jax #bert #feature-extraction #endpoints_compatible #region-us 

passage: TAGS
#transformers #pytorch #jax #bert #feature-extraction #endpoints_compatible #region-us 
