We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Loading config file bug
Hi guys.

I have released a new (XLNet based) transformer model for low-resource language Tigrinya 
 [(TigXLNet)](https://github.com/abryeemessi/Transferring-Monolingual-Model-to-Low-Resource-Language)  and found a bug when loading a pre-trained config file: 

My config file looks like:
https://s3.amazonaws.com/models.huggingface.co/bert/abryee/TigXLNet/config.json


       config = AutoConfig.from_pretrained("abryee/TigXLNet")
       print(config.d_head) #prints 48 even though d_head in the given config file is 64.



--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/transformers/configuration_xlnet.py`
Content:
```
1 # coding=utf-8
2 # Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
3 # Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
4 #
5 # Licensed under the Apache License, Version 2.0 (the "License");
6 # you may not use this file except in compliance with the License.
7 # You may obtain a copy of the License at
8 #
9 #     http://www.apache.org/licenses/LICENSE-2.0
10 #
11 # Unless required by applicable law or agreed to in writing, software
12 # distributed under the License is distributed on an "AS IS" BASIS,
13 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 # See the License for the specific language governing permissions and
15 # limitations under the License.
16 """ XLNet configuration """
17 
18 
19 import logging
20 
21 from .configuration_utils import PretrainedConfig
22 
23 
24 logger = logging.getLogger(__name__)
25 
26 XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP = {
27     "xlnet-base-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-base-cased-config.json",
28     "xlnet-large-cased": "https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-large-cased-config.json",
29 }
30 
31 
32 class XLNetConfig(PretrainedConfig):
33     """
34         This is the configuration class to store the configuration of a :class:`~transformers.XLNetModel`.
35         It is used to instantiate an XLNet model according to the specified arguments, defining the model
36         architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
37         the `xlnet-large-cased <https://huggingface.co/xlnet-large-cased>`__ architecture.
38 
39         Configuration objects inherit from  :class:`~transformers.PretrainedConfig` and can be used
40         to control the model outputs. Read the documentation from  :class:`~transformers.PretrainedConfig`
41         for more information.
42 
43         Args:
44             vocab_size (:obj:`int`, optional, defaults to 32000):
45                 Vocabulary size of the XLNet model. Defines the different tokens that
46                 can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.XLNetModel`.
47             d_model (:obj:`int`, optional, defaults to 1024):
48                 Dimensionality of the encoder layers and the pooler layer.
49             n_layer (:obj:`int`, optional, defaults to 24):
50                 Number of hidden layers in the Transformer encoder.
51             n_head (:obj:`int`, optional, defaults to 16):
52                 Number of attention heads for each attention layer in the Transformer encoder.
53             d_inner (:obj:`int`, optional, defaults to 4096):
54                 Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
55             ff_activation (:obj:`string`, optional, defaults to "gelu"):
56                 The non-linear activation function (function or string) in the
57                 encoder and pooler. If string, "gelu", "relu" and "swish" are supported.
58             untie_r (:obj:`boolean`, optional, defaults to :obj:`True`):
59                 Untie relative position biases
60             attn_type (:obj:`string`, optional, defaults to "bi"):
61                 The attention type used by the model. Set 'bi' for XLNet, 'uni' for Transformer-XL.
62             initializer_range (:obj:`float`, optional, defaults to 0.02):
63                 The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
64             layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
65                 The epsilon used by the layer normalization layers.
66             dropout (:obj:`float`, optional, defaults to 0.1):
67                 The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
68             mem_len (:obj:`int` or :obj:`None`, optional, defaults to :obj:`None`):
69                 The number of tokens to cache. The key/value pairs that have already been pre-computed
70                 in a previous forward pass won't be re-computed. See the
71                 `quickstart <https://huggingface.co/transformers/quickstart.html#using-the-past>`__
72                 for more information.
73             reuse_len (:obj:`int` or :obj:`None`, optional, defaults to :obj:`None`):
74                 The number of tokens in the current batch to be cached and reused in the future.
75             bi_data (:obj:`boolean`, optional, defaults to :obj:`False`):
76                 Whether to use bidirectional input pipeline. Usually set to `True` during
77                 pretraining and `False` during finetuning.
78             clamp_len (:obj:`int`, optional, defaults to -1):
79                 Clamp all relative distances larger than clamp_len.
80                 Setting this attribute to -1 means no clamping.
81             same_length (:obj:`boolean`, optional, defaults to :obj:`False`):
82                 Whether to use the same attention length for each token.
83             summary_type (:obj:`string`, optional, defaults to "last"):
84                 Argument used when doing sequence summary. Used in for the multiple choice head in
85                 :class:transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
86                 Is one of the following options:
87                     - 'last' => take the last token hidden state (like XLNet)
88                     - 'first' => take the first token hidden state (like Bert)
89                     - 'mean' => take the mean of all tokens hidden states
90                     - 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)
91                     - 'attn' => Not implemented now, use multi-head attention
92             summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):
93                 Argument used when doing sequence summary. Used in for the multiple choice head in
94                 :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
95                 Add a projection after the vector extraction
96             summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):
97                 Argument used when doing sequence summary. Used in for the multiple choice head in
98                 :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
99                 'tanh' => add a tanh activation to the output, Other => no activation.
100             summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):
101                 Argument used when doing sequence summary. Used in for the multiple choice head in
102                 :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
103                 If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.
104             summary_last_dropout (:obj:`float`, optional, defaults to 0.1):
105                 Argument used when doing sequence summary. Used in for the multiple choice head in
106                 :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.
107                 Add a dropout after the projection and activation
108             start_n_top (:obj:`int`, optional, defaults to 5):
109                 Used in the SQuAD evaluation script for XLM and XLNet.
110             end_n_top (:obj:`int`, optional, defaults to 5):
111                 Used in the SQuAD evaluation script for XLM and XLNet.
112 
113         Example::
114 
115             from transformers import XLNetConfig, XLNetModel
116 
117             # Initializing a XLNet configuration
118             configuration = XLNetConfig()
119 
120             # Initializing a model from the configuration
121             model = XLNetModel(configuration)
122 
123             # Accessing the model configuration
124             configuration = model.config
125     """
126 
127     model_type = "xlnet"
128 
129     def __init__(
130         self,
131         vocab_size=32000,
132         d_model=1024,
133         n_layer=24,
134         n_head=16,
135         d_inner=4096,
136         ff_activation="gelu",
137         untie_r=True,
138         attn_type="bi",
139         initializer_range=0.02,
140         layer_norm_eps=1e-12,
141         dropout=0.1,
142         mem_len=None,
143         reuse_len=None,
144         bi_data=False,
145         clamp_len=-1,
146         same_length=False,
147         summary_type="last",
148         summary_use_proj=True,
149         summary_activation="tanh",
150         summary_last_dropout=0.1,
151         start_n_top=5,
152         end_n_top=5,
153         pad_token_id=5,
154         bos_token_id=1,
155         eos_token_id=2,
156         **kwargs
157     ):
158         """Constructs XLNetConfig.
159         """
160         super().__init__(pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)
161         self.vocab_size = vocab_size
162         self.d_model = d_model
163         self.n_layer = n_layer
164         self.n_head = n_head
165         assert d_model % n_head == 0
166         self.d_head = d_model // n_head
167         self.ff_activation = ff_activation
168         self.d_inner = d_inner
169         self.untie_r = untie_r
170         self.attn_type = attn_type
171 
172         self.initializer_range = initializer_range
173         self.layer_norm_eps = layer_norm_eps
174 
175         self.dropout = dropout
176         self.mem_len = mem_len
177         self.reuse_len = reuse_len
178         self.bi_data = bi_data
179         self.clamp_len = clamp_len
180         self.same_length = same_length
181 
182         self.summary_type = summary_type
183         self.summary_use_proj = summary_use_proj
184         self.summary_activation = summary_activation
185         self.summary_last_dropout = summary_last_dropout
186         self.start_n_top = start_n_top
187         self.end_n_top = end_n_top
188 
189         self.bos_token_id = bos_token_id
190         self.pad_token_id = pad_token_id
191         self.eos_token_id = eos_token_id
192 
193     @property
194     def max_position_embeddings(self):
195         return -1
196 
197     @property
198     def n_token(self):  # Backward compatibility
199         return self.vocab_size
200 
201     @n_token.setter
202     def n_token(self, value):  # Backward compatibility
203         self.vocab_size = value
204 
205     @property
206     def hidden_size(self):
207         return self.d_model
208 
209     @property
210     def num_attention_heads(self):
211         return self.n_head
212 
213     @property
214     def num_hidden_layers(self):
215         return self.n_layer
216
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/transformers/configuration_xlnet.py b/src/transformers/configuration_xlnet.py
--- a/src/transformers/configuration_xlnet.py
+++ b/src/transformers/configuration_xlnet.py
@@ -163,6 +163,10 @@
         self.n_layer = n_layer
         self.n_head = n_head
         assert d_model % n_head == 0
+        if "d_head" in kwargs:
+            assert kwargs["d_head"] == d_model // n_head, (
+                f"`d_head` ({kwargs['d_head']}) should be equal to `d_model // n_head` ({d_model // n_head})"
+            )
         self.d_head = d_model // n_head
         self.ff_activation = ff_activation
         self.d_inner = d_inner

{"golden_diff": "diff --git a/src/transformers/configuration_xlnet.py b/src/transformers/configuration_xlnet.py\n--- a/src/transformers/configuration_xlnet.py\n+++ b/src/transformers/configuration_xlnet.py\n@@ -163,6 +163,10 @@\n         self.n_layer = n_layer\n         self.n_head = n_head\n         assert d_model % n_head == 0\n+        if \"d_head\" in kwargs:\n+            assert kwargs[\"d_head\"] == d_model // n_head, (\n+                f\"`d_head` ({kwargs['d_head']}) should be equal to `d_model // n_head` ({d_model // n_head})\"\n+            )\n         self.d_head = d_model // n_head\n         self.ff_activation = ff_activation\n         self.d_inner = d_inner\n", "issue": "Loading config file bug\nHi guys.\r\n\r\nI have released a new (XLNet based) transformer model for low-resource language Tigrinya \r\n [(TigXLNet)](https://github.com/abryeemessi/Transferring-Monolingual-Model-to-Low-Resource-Language)  and found a bug when loading a pre-trained config file: \r\n\r\nMy config file looks like:\r\nhttps://s3.amazonaws.com/models.huggingface.co/bert/abryee/TigXLNet/config.json\r\n\r\n\r\n       config = AutoConfig.from_pretrained(\"abryee/TigXLNet\")\r\n       print(config.d_head) #prints 48 even though d_head in the given config file is 64.\r\n\r\n\n", "before_files": [{"content": "# coding=utf-8\n# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.\n# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\" XLNet configuration \"\"\"\n\n\nimport logging\n\nfrom .configuration_utils import PretrainedConfig\n\n\nlogger = logging.getLogger(__name__)\n\nXLNET_PRETRAINED_CONFIG_ARCHIVE_MAP = {\n    \"xlnet-base-cased\": \"https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-base-cased-config.json\",\n    \"xlnet-large-cased\": \"https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-large-cased-config.json\",\n}\n\n\nclass XLNetConfig(PretrainedConfig):\n    \"\"\"\n        This is the configuration class to store the configuration of a :class:`~transformers.XLNetModel`.\n        It is used to instantiate an XLNet model according to the specified arguments, defining the model\n        architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of\n        the `xlnet-large-cased <https://huggingface.co/xlnet-large-cased>`__ architecture.\n\n        Configuration objects inherit from  :class:`~transformers.PretrainedConfig` and can be used\n        to control the model outputs. Read the documentation from  :class:`~transformers.PretrainedConfig`\n        for more information.\n\n        Args:\n            vocab_size (:obj:`int`, optional, defaults to 32000):\n                Vocabulary size of the XLNet model. Defines the different tokens that\n                can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.XLNetModel`.\n            d_model (:obj:`int`, optional, defaults to 1024):\n                Dimensionality of the encoder layers and the pooler layer.\n            n_layer (:obj:`int`, optional, defaults to 24):\n                Number of hidden layers in the Transformer encoder.\n            n_head (:obj:`int`, optional, defaults to 16):\n                Number of attention heads for each attention layer in the Transformer encoder.\n            d_inner (:obj:`int`, optional, defaults to 4096):\n                Dimensionality of the \"intermediate\" (i.e., feed-forward) layer in the Transformer encoder.\n            ff_activation (:obj:`string`, optional, defaults to \"gelu\"):\n                The non-linear activation function (function or string) in the\n                encoder and pooler. If string, \"gelu\", \"relu\" and \"swish\" are supported.\n            untie_r (:obj:`boolean`, optional, defaults to :obj:`True`):\n                Untie relative position biases\n            attn_type (:obj:`string`, optional, defaults to \"bi\"):\n                The attention type used by the model. Set 'bi' for XLNet, 'uni' for Transformer-XL.\n            initializer_range (:obj:`float`, optional, defaults to 0.02):\n                The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\n            layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):\n                The epsilon used by the layer normalization layers.\n            dropout (:obj:`float`, optional, defaults to 0.1):\n                The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\n            mem_len (:obj:`int` or :obj:`None`, optional, defaults to :obj:`None`):\n                The number of tokens to cache. The key/value pairs that have already been pre-computed\n                in a previous forward pass won't be re-computed. See the\n                `quickstart <https://huggingface.co/transformers/quickstart.html#using-the-past>`__\n                for more information.\n            reuse_len (:obj:`int` or :obj:`None`, optional, defaults to :obj:`None`):\n                The number of tokens in the current batch to be cached and reused in the future.\n            bi_data (:obj:`boolean`, optional, defaults to :obj:`False`):\n                Whether to use bidirectional input pipeline. Usually set to `True` during\n                pretraining and `False` during finetuning.\n            clamp_len (:obj:`int`, optional, defaults to -1):\n                Clamp all relative distances larger than clamp_len.\n                Setting this attribute to -1 means no clamping.\n            same_length (:obj:`boolean`, optional, defaults to :obj:`False`):\n                Whether to use the same attention length for each token.\n            summary_type (:obj:`string`, optional, defaults to \"last\"):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                Is one of the following options:\n                    - 'last' => take the last token hidden state (like XLNet)\n                    - 'first' => take the first token hidden state (like Bert)\n                    - 'mean' => take the mean of all tokens hidden states\n                    - 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)\n                    - 'attn' => Not implemented now, use multi-head attention\n            summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                Add a projection after the vector extraction\n            summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                'tanh' => add a tanh activation to the output, Other => no activation.\n            summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.\n            summary_last_dropout (:obj:`float`, optional, defaults to 0.1):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                Add a dropout after the projection and activation\n            start_n_top (:obj:`int`, optional, defaults to 5):\n                Used in the SQuAD evaluation script for XLM and XLNet.\n            end_n_top (:obj:`int`, optional, defaults to 5):\n                Used in the SQuAD evaluation script for XLM and XLNet.\n\n        Example::\n\n            from transformers import XLNetConfig, XLNetModel\n\n            # Initializing a XLNet configuration\n            configuration = XLNetConfig()\n\n            # Initializing a model from the configuration\n            model = XLNetModel(configuration)\n\n            # Accessing the model configuration\n            configuration = model.config\n    \"\"\"\n\n    model_type = \"xlnet\"\n\n    def __init__(\n        self,\n        vocab_size=32000,\n        d_model=1024,\n        n_layer=24,\n        n_head=16,\n        d_inner=4096,\n        ff_activation=\"gelu\",\n        untie_r=True,\n        attn_type=\"bi\",\n        initializer_range=0.02,\n        layer_norm_eps=1e-12,\n        dropout=0.1,\n        mem_len=None,\n        reuse_len=None,\n        bi_data=False,\n        clamp_len=-1,\n        same_length=False,\n        summary_type=\"last\",\n        summary_use_proj=True,\n        summary_activation=\"tanh\",\n        summary_last_dropout=0.1,\n        start_n_top=5,\n        end_n_top=5,\n        pad_token_id=5,\n        bos_token_id=1,\n        eos_token_id=2,\n        **kwargs\n    ):\n        \"\"\"Constructs XLNetConfig.\n        \"\"\"\n        super().__init__(pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)\n        self.vocab_size = vocab_size\n        self.d_model = d_model\n        self.n_layer = n_layer\n        self.n_head = n_head\n        assert d_model % n_head == 0\n        self.d_head = d_model // n_head\n        self.ff_activation = ff_activation\n        self.d_inner = d_inner\n        self.untie_r = untie_r\n        self.attn_type = attn_type\n\n        self.initializer_range = initializer_range\n        self.layer_norm_eps = layer_norm_eps\n\n        self.dropout = dropout\n        self.mem_len = mem_len\n        self.reuse_len = reuse_len\n        self.bi_data = bi_data\n        self.clamp_len = clamp_len\n        self.same_length = same_length\n\n        self.summary_type = summary_type\n        self.summary_use_proj = summary_use_proj\n        self.summary_activation = summary_activation\n        self.summary_last_dropout = summary_last_dropout\n        self.start_n_top = start_n_top\n        self.end_n_top = end_n_top\n\n        self.bos_token_id = bos_token_id\n        self.pad_token_id = pad_token_id\n        self.eos_token_id = eos_token_id\n\n    @property\n    def max_position_embeddings(self):\n        return -1\n\n    @property\n    def n_token(self):  # Backward compatibility\n        return self.vocab_size\n\n    @n_token.setter\n    def n_token(self, value):  # Backward compatibility\n        self.vocab_size = value\n\n    @property\n    def hidden_size(self):\n        return self.d_model\n\n    @property\n    def num_attention_heads(self):\n        return self.n_head\n\n    @property\n    def num_hidden_layers(self):\n        return self.n_layer\n", "path": "src/transformers/configuration_xlnet.py"}], "after_files": [{"content": "# coding=utf-8\n# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.\n# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\" XLNet configuration \"\"\"\n\n\nimport logging\n\nfrom .configuration_utils import PretrainedConfig\n\n\nlogger = logging.getLogger(__name__)\n\nXLNET_PRETRAINED_CONFIG_ARCHIVE_MAP = {\n    \"xlnet-base-cased\": \"https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-base-cased-config.json\",\n    \"xlnet-large-cased\": \"https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-large-cased-config.json\",\n}\n\n\nclass XLNetConfig(PretrainedConfig):\n    \"\"\"\n        This is the configuration class to store the configuration of a :class:`~transformers.XLNetModel`.\n        It is used to instantiate an XLNet model according to the specified arguments, defining the model\n        architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of\n        the `xlnet-large-cased <https://huggingface.co/xlnet-large-cased>`__ architecture.\n\n        Configuration objects inherit from  :class:`~transformers.PretrainedConfig` and can be used\n        to control the model outputs. Read the documentation from  :class:`~transformers.PretrainedConfig`\n        for more information.\n\n        Args:\n            vocab_size (:obj:`int`, optional, defaults to 32000):\n                Vocabulary size of the XLNet model. Defines the different tokens that\n                can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.XLNetModel`.\n            d_model (:obj:`int`, optional, defaults to 1024):\n                Dimensionality of the encoder layers and the pooler layer.\n            n_layer (:obj:`int`, optional, defaults to 24):\n                Number of hidden layers in the Transformer encoder.\n            n_head (:obj:`int`, optional, defaults to 16):\n                Number of attention heads for each attention layer in the Transformer encoder.\n            d_inner (:obj:`int`, optional, defaults to 4096):\n                Dimensionality of the \"intermediate\" (i.e., feed-forward) layer in the Transformer encoder.\n            ff_activation (:obj:`string`, optional, defaults to \"gelu\"):\n                The non-linear activation function (function or string) in the\n                encoder and pooler. If string, \"gelu\", \"relu\" and \"swish\" are supported.\n            untie_r (:obj:`boolean`, optional, defaults to :obj:`True`):\n                Untie relative position biases\n            attn_type (:obj:`string`, optional, defaults to \"bi\"):\n                The attention type used by the model. Set 'bi' for XLNet, 'uni' for Transformer-XL.\n            initializer_range (:obj:`float`, optional, defaults to 0.02):\n                The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\n            layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):\n                The epsilon used by the layer normalization layers.\n            dropout (:obj:`float`, optional, defaults to 0.1):\n                The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\n            mem_len (:obj:`int` or :obj:`None`, optional, defaults to :obj:`None`):\n                The number of tokens to cache. The key/value pairs that have already been pre-computed\n                in a previous forward pass won't be re-computed. See the\n                `quickstart <https://huggingface.co/transformers/quickstart.html#using-the-past>`__\n                for more information.\n            reuse_len (:obj:`int` or :obj:`None`, optional, defaults to :obj:`None`):\n                The number of tokens in the current batch to be cached and reused in the future.\n            bi_data (:obj:`boolean`, optional, defaults to :obj:`False`):\n                Whether to use bidirectional input pipeline. Usually set to `True` during\n                pretraining and `False` during finetuning.\n            clamp_len (:obj:`int`, optional, defaults to -1):\n                Clamp all relative distances larger than clamp_len.\n                Setting this attribute to -1 means no clamping.\n            same_length (:obj:`boolean`, optional, defaults to :obj:`False`):\n                Whether to use the same attention length for each token.\n            summary_type (:obj:`string`, optional, defaults to \"last\"):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                Is one of the following options:\n                    - 'last' => take the last token hidden state (like XLNet)\n                    - 'first' => take the first token hidden state (like Bert)\n                    - 'mean' => take the mean of all tokens hidden states\n                    - 'cls_index' => supply a Tensor of classification token position (GPT/GPT-2)\n                    - 'attn' => Not implemented now, use multi-head attention\n            summary_use_proj (:obj:`boolean`, optional, defaults to :obj:`True`):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                Add a projection after the vector extraction\n            summary_activation (:obj:`string` or :obj:`None`, optional, defaults to :obj:`None`):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                'tanh' => add a tanh activation to the output, Other => no activation.\n            summary_proj_to_labels (:obj:`boolean`, optional, defaults to :obj:`True`):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                If True, the projection outputs to config.num_labels classes (otherwise to hidden_size). Default: False.\n            summary_last_dropout (:obj:`float`, optional, defaults to 0.1):\n                Argument used when doing sequence summary. Used in for the multiple choice head in\n                :class:`~transformers.XLNetForSequenceClassification` and :class:`~transformers.XLNetForMultipleChoice`.\n                Add a dropout after the projection and activation\n            start_n_top (:obj:`int`, optional, defaults to 5):\n                Used in the SQuAD evaluation script for XLM and XLNet.\n            end_n_top (:obj:`int`, optional, defaults to 5):\n                Used in the SQuAD evaluation script for XLM and XLNet.\n\n        Example::\n\n            from transformers import XLNetConfig, XLNetModel\n\n            # Initializing a XLNet configuration\n            configuration = XLNetConfig()\n\n            # Initializing a model from the configuration\n            model = XLNetModel(configuration)\n\n            # Accessing the model configuration\n            configuration = model.config\n    \"\"\"\n\n    model_type = \"xlnet\"\n\n    def __init__(\n        self,\n        vocab_size=32000,\n        d_model=1024,\n        n_layer=24,\n        n_head=16,\n        d_inner=4096,\n        ff_activation=\"gelu\",\n        untie_r=True,\n        attn_type=\"bi\",\n        initializer_range=0.02,\n        layer_norm_eps=1e-12,\n        dropout=0.1,\n        mem_len=None,\n        reuse_len=None,\n        bi_data=False,\n        clamp_len=-1,\n        same_length=False,\n        summary_type=\"last\",\n        summary_use_proj=True,\n        summary_activation=\"tanh\",\n        summary_last_dropout=0.1,\n        start_n_top=5,\n        end_n_top=5,\n        pad_token_id=5,\n        bos_token_id=1,\n        eos_token_id=2,\n        **kwargs\n    ):\n        \"\"\"Constructs XLNetConfig.\n        \"\"\"\n        super().__init__(pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs)\n        self.vocab_size = vocab_size\n        self.d_model = d_model\n        self.n_layer = n_layer\n        self.n_head = n_head\n        assert d_model % n_head == 0\n        if \"d_head\" in kwargs:\n            assert kwargs[\"d_head\"] == d_model // n_head, (\n                f\"`d_head` ({kwargs['d_head']}) should be equal to `d_model // n_head` ({d_model // n_head})\"\n            )\n        self.d_head = d_model // n_head\n        self.ff_activation = ff_activation\n        self.d_inner = d_inner\n        self.untie_r = untie_r\n        self.attn_type = attn_type\n\n        self.initializer_range = initializer_range\n        self.layer_norm_eps = layer_norm_eps\n\n        self.dropout = dropout\n        self.mem_len = mem_len\n        self.reuse_len = reuse_len\n        self.bi_data = bi_data\n        self.clamp_len = clamp_len\n        self.same_length = same_length\n\n        self.summary_type = summary_type\n        self.summary_use_proj = summary_use_proj\n        self.summary_activation = summary_activation\n        self.summary_last_dropout = summary_last_dropout\n        self.start_n_top = start_n_top\n        self.end_n_top = end_n_top\n\n        self.bos_token_id = bos_token_id\n        self.pad_token_id = pad_token_id\n        self.eos_token_id = eos_token_id\n\n    @property\n    def max_position_embeddings(self):\n        return -1\n\n    @property\n    def n_token(self):  # Backward compatibility\n        return self.vocab_size\n\n    @n_token.setter\n    def n_token(self, value):  # Backward compatibility\n        self.vocab_size = value\n\n    @property\n    def hidden_size(self):\n        return self.d_model\n\n    @property\n    def num_attention_heads(self):\n        return self.n_head\n\n    @property\n    def num_hidden_layers(self):\n        return self.n_layer\n", "path": "src/transformers/configuration_xlnet.py"}]}