

# viniLM-2021-from-large

### Model Info:
<ul>
<li> Authors: R&D AI Lab, VMware Inc.
<li> Model date: Jun 2022
<li> Model version: 2021-distilled-from-large
<li> Model type: Pretrained language model
<li> License: Apache 2.0
</ul>

#### Motivation
Based on [MiniLMv2 distillation](https://arxiv.org/pdf/2012.15828.pdf), we have distilled vBERT-2021-large into a smaller minilmv2 model for faster inference times without a significant loss of performance. 

#### Intended Use
The model functions as a VMware-specific Language Model.


#### How to Use
Here is how to use this model to get the features of a given text in PyTorch:

```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('VMware/vinilm-2021-from-large')
model = BertModel.from_pretrained("VMware/vinilm-2021-from-large")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

and in TensorFlow:

```
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('VMware/vinilm-2021-from-large')
model = TFBertModel.from_pretrained('VMware/vinilm-2021-from-large')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

```

### Training

The model is distilled from [vBERT-2021-large](https://huggingface.co/VMware/vbert-2021-large)</li> <br>
Weights were initialized using [nreimers/MiniLMv2-L6-H768-distilled-from-BERT-Large](https://huggingface.co/nreimers/MiniLMv2-L6-H768-distilled-from-BERT-Large/tree/main)</li>



#### - Datasets
Publically available VMware text data such as VMware Docs, Blogs, etc. were used for distilling the teacher vBERT-2021-large model into vinilm-2021-from-large model. Sourced in May 2021. (~320,000 Documents)
#### - Preprocessing
<ul>
<li>Decoding HTML
<li>Decoding Unicode
<li>Stripping repeated characters
<li>Splitting compound word
<li>Spelling correction
</ul>

#### - Model performance measures
We benchmarked vinilm on various VMware-specific NLP downstream tasks (IR, classification, etc).

### Limitations and bias
Since the model is distilled from a vBERT model based on the BERT model, it may have the same biases embedded within the original BERT model.

The data needs to be preprocessed using our internal vNLP Preprocessor (not available to the public) to maximize its performance.

{"language": ["eng"], "license": "apache-2.0", "tags": ["PyTorch", "tensorflow"], "thumbnail": "URL to a thumbnail used in social sharing"}
transformers pytorch bert feature-extraction PyTorch tensorflow eng arxiv:2012.15828 license:apache-2.0 endpoints_compatible region:us
TAGS
#transformers #pytorch #bert #feature-extraction #PyTorch #tensorflow #eng #arxiv-2012.15828 #license-apache-2.0 #endpoints_compatible #region-us 



# viniLM-2021-from-large

### Model Info:
<ul>
<li> Authors: R&D AI Lab, VMware Inc.
<li> Model date: Jun 2022
<li> Model version: 2021-distilled-from-large
<li> Model type: Pretrained language model
<li> License: Apache 2.0
</ul>

#### Motivation
Based on MiniLMv2 distillation, we have distilled vBERT-2021-large into a smaller minilmv2 model for faster inference times without a significant loss of performance. 

#### Intended Use
The model functions as a VMware-specific Language Model.


#### How to Use
Here is how to use this model to get the features of a given text in PyTorch:



and in TensorFlow:



### Training

The model is distilled from vBERT-2021-large</li> <br>
Weights were initialized using nreimers/MiniLMv2-L6-H768-distilled-from-BERT-Large</li>



#### - Datasets
Publically available VMware text data such as VMware Docs, Blogs, etc. were used for distilling the teacher vBERT-2021-large model into vinilm-2021-from-large model. Sourced in May 2021. (~320,000 Documents)
#### - Preprocessing
<ul>
<li>Decoding HTML
<li>Decoding Unicode
<li>Stripping repeated characters
<li>Splitting compound word
<li>Spelling correction
</ul>

#### - Model performance measures
We benchmarked vinilm on various VMware-specific NLP downstream tasks (IR, classification, etc).

### Limitations and bias
Since the model is distilled from a vBERT model based on the BERT model, it may have the same biases embedded within the original BERT model.

The data needs to be preprocessed using our internal vNLP Preprocessor (not available to the public) to maximize its performance.

# viniLM-2021-from-large ### Model Info:
<ul>
<li> Authors: R&D AI Lab, VMware Inc.
<li> Model date: Jun 2022
<li> Model version: 2021-distilled-from-large
<li> Model type: Pretrained language model
<li> License: Apache 2.0
</ul> #### Motivation
Based on MiniLMv2 distillation, we have distilled vBERT-2021-large into a smaller minilmv2 model for faster inference times without a significant loss of performance. #### Intended Use
The model functions as a VMware-specific Language Model. #### How to Use
Here is how to use this model to get the features of a given text in PyTorch:



and in TensorFlow: ### Training

The model is distilled from vBERT-2021-large</li> <br>
Weights were initialized using nreimers/MiniLMv2-L6-H768-distilled-from-BERT-Large</li> #### - Datasets
Publically available VMware text data such as VMware Docs, Blogs, etc. were used for distilling the teacher vBERT-2021-large model into vinilm-2021-from-large model. Sourced in May 2021. (~320,000 Documents) #### - Preprocessing
<ul>
<li>Decoding HTML
<li>Decoding Unicode
<li>Stripping repeated characters
<li>Splitting compound word
<li>Spelling correction
</ul> #### - Model performance measures
We benchmarked vinilm on various VMware-specific NLP downstream tasks (IR, classification, etc). ### Limitations and bias
Since the model is distilled from a vBERT model based on the BERT model, it may have the same biases embedded within the original BERT model.

The data needs to be preprocessed using our internal vNLP Preprocessor (not available to the public) to maximize its performance.
TAGS
#transformers #pytorch #bert #feature-extraction #PyTorch #tensorflow #eng #arxiv-2012.15828 #license-apache-2.0 #endpoints_compatible #region-us 
 # viniLM-2021-from-large ### Model Info:
<ul>
<li> Authors: R&D AI Lab, VMware Inc.
<li> Model date: Jun 2022
<li> Model version: 2021-distilled-from-large
<li> Model type: Pretrained language model
<li> License: Apache 2.0
</ul> #### Motivation
Based on MiniLMv2 distillation, we have distilled vBERT-2021-large into a smaller minilmv2 model for faster inference times without a significant loss of performance. #### Intended Use
The model functions as a VMware-specific Language Model. #### How to Use
Here is how to use this model to get the features of a given text in PyTorch:



and in TensorFlow: ### Training

The model is distilled from vBERT-2021-large</li> <br>
Weights were initialized using nreimers/MiniLMv2-L6-H768-distilled-from-BERT-Large</li> #### - Datasets
Publically available VMware text data such as VMware Docs, Blogs, etc. were used for distilling the teacher vBERT-2021-large model into vinilm-2021-from-large model. Sourced in May 2021. (~320,000 Documents) #### - Preprocessing
<ul>
<li>Decoding HTML
<li>Decoding Unicode
<li>Stripping repeated characters
<li>Splitting compound word
<li>Spelling correction
</ul> #### - Model performance measures
We benchmarked vinilm on various VMware-specific NLP downstream tasks (IR, classification, etc). ### Limitations and bias
Since the model is distilled from a vBERT model based on the BERT model, it may have the same biases embedded within the original BERT model.

The data needs to be preprocessed using our internal vNLP Preprocessor (not available to the public) to maximize its performance.
passage: TAGS
#transformers #pytorch #bert #feature-extraction #PyTorch #tensorflow #eng #arxiv-2012.15828 #license-apache-2.0 #endpoints_compatible #region-us 
# viniLM-2021-from-large### Model Info:
<ul>
<li> Authors: R&D AI Lab, VMware Inc.
<li> Model date: Jun 2022
<li> Model version: 2021-distilled-from-large
<li> Model type: Pretrained language model
<li> License: Apache 2.0
</ul>#### Motivation
Based on MiniLMv2 distillation, we have distilled vBERT-2021-large into a smaller minilmv2 model for faster inference times without a significant loss of performance.#### Intended Use
The model functions as a VMware-specific Language Model.#### How to Use
Here is how to use this model to get the features of a given text in PyTorch:



and in TensorFlow:### Training

The model is distilled from vBERT-2021-large</li> <br>
Weights were initialized using nreimers/MiniLMv2-L6-H768-distilled-from-BERT-Large</li>#### - Datasets
Publically available VMware text data such as VMware Docs, Blogs, etc. were used for distilling the teacher vBERT-2021-large model into vinilm-2021-from-large model. Sourced in May 2021. (~320,000 Documents)#### - Preprocessing
<ul>
<li>Decoding HTML
<li>Decoding Unicode
<li>Stripping repeated characters
<li>Splitting compound word
<li>Spelling correction
</ul>#### - Model performance measures
We benchmarked vinilm on various VMware-specific NLP downstream tasks (IR, classification, etc).