# DistilBERT with 256k token embeddings

This model was initialized with a word2vec token embedding matrix with 256k entries, but these token embeddings were updated during MLM. The word2vec was trained on 100GB data from C4, MSMARCO, News, Wikipedia, S2ORC, for 3 epochs.

Then the model was trained on this dataset with MLM for 750k steps (batch size 64). The token embeddings were updated during MLM.

For the same model but with frozen token embeddings while MLM training see: https://huggingface.co/vocab-transformers/distilbert-word2vec_256k-MLM_750k


{}
transformers pytorch distilbert fill-mask autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #distilbert #fill-mask #autotrain_compatible #endpoints_compatible #region-us 

# DistilBERT with 256k token embeddings

This model was initialized with a word2vec token embedding matrix with 256k entries, but these token embeddings were updated during MLM. The word2vec was trained on 100GB data from C4, MSMARCO, News, Wikipedia, S2ORC, for 3 epochs.

Then the model was trained on this dataset with MLM for 750k steps (batch size 64). The token embeddings were updated during MLM.

For the same model but with frozen token embeddings while MLM training see: URL


# DistilBERT with 256k token embeddings

This model was initialized with a word2vec token embedding matrix with 256k entries, but these token embeddings were updated during MLM. The word2vec was trained on 100GB data from C4, MSMARCO, News, Wikipedia, S2ORC, for 3 epochs.

Then the model was trained on this dataset with MLM for 750k steps (batch size 64). The token embeddings were updated during MLM.

For the same model but with frozen token embeddings while MLM training see: URL
TAGS
#transformers #pytorch #distilbert #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
 # DistilBERT with 256k token embeddings

This model was initialized with a word2vec token embedding matrix with 256k entries, but these token embeddings were updated during MLM. The word2vec was trained on 100GB data from C4, MSMARCO, News, Wikipedia, S2ORC, for 3 epochs.

Then the model was trained on this dataset with MLM for 750k steps (batch size 64). The token embeddings were updated during MLM.

For the same model but with frozen token embeddings while MLM training see: URL
passage: TAGS
#transformers #pytorch #distilbert #fill-mask #autotrain_compatible #endpoints_compatible #region-us 
# DistilBERT with 256k token embeddings

This model was initialized with a word2vec token embedding matrix with 256k entries, but these token embeddings were updated during MLM. The word2vec was trained on 100GB data from C4, MSMARCO, News, Wikipedia, S2ORC, for 3 epochs.

Then the model was trained on this dataset with MLM for 750k steps (batch size 64). The token embeddings were updated during MLM.

For the same model but with frozen token embeddings while MLM training see: URL