
## mLUKE

**mLUKE** (multilingual LUKE) is a multilingual extension of LUKE.

Please check the [official repository](https://github.com/studio-ousia/luke) for
more details and updates.

This is the mLUKE large model with 24 hidden layers, 768 hidden size. The total number
of parameters in this model is 868M (561M for the word embeddings and encoder, 307M for the entity embeddings).
The model was initialized with the weights of XLM-RoBERTa(large) and trained using December 2020 version of Wikipedia in 24 languages.

## Note
When you load the model from `AutoModel.from_pretrained` with the default configuration, you will see the following warning:

```
Some weights of the model checkpoint at studio-ousia/mluke-base-lite were not used when initializing LukeModel: [
'luke.encoder.layer.0.attention.self.w2e_query.weight', 'luke.encoder.layer.0.attention.self.w2e_query.bias', 
'luke.encoder.layer.0.attention.self.e2w_query.weight', 'luke.encoder.layer.0.attention.self.e2w_query.bias', 
'luke.encoder.layer.0.attention.self.e2e_query.weight', 'luke.encoder.layer.0.attention.self.e2e_query.bias', 
...]
```

These weights are the weights for entity-aware attention (as described in [the LUKE paper](https://arxiv.org/abs/2010.01057)).
This is expected because `use_entity_aware_attention` is set to `false` by default, but the pretrained weights contain the weights for it in case you enable `use_entity_aware_attention` and have the weights loaded into the model.


### Citation

If you find mLUKE useful for your work, please cite the following paper:

```latex
@inproceedings{ri-etal-2022-mluke,
    title = "m{LUKE}: {T}he Power of Entity Representations in Multilingual Pretrained Language Models",
    author = "Ri, Ryokan  and
      Yamada, Ikuya  and
      Tsuruoka, Yoshimasa",
    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    year = "2022",
    url = "https://aclanthology.org/2022.acl-long.505",
```
{"language": ["multilingual", "ar", "bn", "de", "el", "en", "es", "fi", "fr", "hi", "id", "it", "ja", "ko", "nl", "pl", "pt", "ru", "sv", "sw", "te", "th", "tr", "vi", "zh"], "license": "apache-2.0", "tags": ["luke", "named entity recognition", "relation classification", "question answering"], "thumbnail": "https://github.com/studio-ousia/luke/raw/master/resources/luke_logo.png"}
transformers pytorch luke fill-mask named entity recognition relation classification question answering multilingual ar bn de el en es fi fr hi id it ja ko nl pl pt ru sv sw te th tr vi zh arxiv:2010.01057 license:apache-2.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #luke #fill-mask #named entity recognition #relation classification #question answering #multilingual #ar #bn #de #el #en #es #fi #fr #hi #id #it #ja #ko #nl #pl #pt #ru #sv #sw #te #th #tr #vi #zh #arxiv-2010.01057 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 


## mLUKE

mLUKE (multilingual LUKE) is a multilingual extension of LUKE.

Please check the official repository for
more details and updates.

This is the mLUKE large model with 24 hidden layers, 768 hidden size. The total number
of parameters in this model is 868M (561M for the word embeddings and encoder, 307M for the entity embeddings).
The model was initialized with the weights of XLM-RoBERTa(large) and trained using December 2020 version of Wikipedia in 24 languages.

## Note
When you load the model from 'AutoModel.from_pretrained' with the default configuration, you will see the following warning:



These weights are the weights for entity-aware attention (as described in the LUKE paper).
This is expected because 'use_entity_aware_attention' is set to 'false' by default, but the pretrained weights contain the weights for it in case you enable 'use_entity_aware_attention' and have the weights loaded into the model.


If you find mLUKE useful for your work, please cite the following paper:


## mLUKE

mLUKE (multilingual LUKE) is a multilingual extension of LUKE.

Please check the official repository for
more details and updates.

This is the mLUKE large model with 24 hidden layers, 768 hidden size. The total number
of parameters in this model is 868M (561M for the word embeddings and encoder, 307M for the entity embeddings).
The model was initialized with the weights of XLM-RoBERTa(large) and trained using December 2020 version of Wikipedia in 24 languages. ## Note
When you load the model from 'AutoModel.from_pretrained' with the default configuration, you will see the following warning:



These weights are the weights for entity-aware attention (as described in the LUKE paper).
This is expected because 'use_entity_aware_attention' is set to 'false' by default, but the pretrained weights contain the weights for it in case you enable 'use_entity_aware_attention' and have the weights loaded into the model.


If you find mLUKE useful for your work, please cite the following paper:
TAGS
#transformers #pytorch #luke #fill-mask #named entity recognition #relation classification #question answering #multilingual #ar #bn #de #el #en #es #fi #fr #hi #id #it #ja #ko #nl #pl #pt #ru #sv #sw #te #th #tr #vi #zh #arxiv-2010.01057 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
 ## mLUKE

mLUKE (multilingual LUKE) is a multilingual extension of LUKE.

Please check the official repository for
more details and updates.

This is the mLUKE large model with 24 hidden layers, 768 hidden size. The total number
of parameters in this model is 868M (561M for the word embeddings and encoder, 307M for the entity embeddings).
The model was initialized with the weights of XLM-RoBERTa(large) and trained using December 2020 version of Wikipedia in 24 languages. ## Note
When you load the model from 'AutoModel.from_pretrained' with the default configuration, you will see the following warning:



These weights are the weights for entity-aware attention (as described in the LUKE paper).
This is expected because 'use_entity_aware_attention' is set to 'false' by default, but the pretrained weights contain the weights for it in case you enable 'use_entity_aware_attention' and have the weights loaded into the model.


If you find mLUKE useful for your work, please cite the following paper:
passage: TAGS
#transformers #pytorch #luke #fill-mask #named entity recognition #relation classification #question answering #multilingual #ar #bn #de #el #en #es #fi #fr #hi #id #it #ja #ko #nl #pl #pt #ru #sv #sw #te #th #tr #vi #zh #arxiv-2010.01057 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
## mLUKE

mLUKE (multilingual LUKE) is a multilingual extension of LUKE.

Please check the official repository for
more details and updates.

This is the mLUKE large model with 24 hidden layers, 768 hidden size. The total number
of parameters in this model is 868M (561M for the word embeddings and encoder, 307M for the entity embeddings).
The model was initialized with the weights of XLM-RoBERTa(large) and trained using December 2020 version of Wikipedia in 24 languages.## Note
When you load the model from 'AutoModel.from_pretrained' with the default configuration, you will see the following warning:



These weights are the weights for entity-aware attention (as described in the LUKE paper).
This is expected because 'use_entity_aware_attention' is set to 'false' by default, but the pretrained weights contain the weights for it in case you enable 'use_entity_aware_attention' and have the weights loaded into the model.


If you find mLUKE useful for your work, please cite the following paper: