Huggingface Electra - Load model trained with google implementation error: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte
I have trained an electra model from scratch using google implementation code.
python run_pretraining.py --data-dir gc://bucket-electra/dataset/ --model-name greek_electra --hparams hparams.json

with this json hyperparams:
{
&quot;embedding_size&quot;: 768,
&quot;max_seq_length&quot;: 512,
&quot;train_batch_size&quot;: 128,
&quot;vocab_size&quot;: 100000,
&quot;model_size&quot;: &quot;base&quot;,
&quot;num_train_steps&quot;: 1500000
}

After having trained the model, I used the convert_electra_original_tf_checkpoint_to_pytorch.py script from transformers library to convert the checkpoint.
python convert_electra_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path output/models/transformer/greek_electra --config_file resources/hparams.json --pytorch_dump_path output/models/transformer/discriminator  --discriminator_or_generator &quot;discriminator&quot;

Now I am trying to load the model:
from transformers import ElectraForPreTraining

model = ElectraForPreTraining.from_pretrained('discriminator')

but I get the following error:
Traceback (most recent call last):
  File &quot;~/.local/lib/python3.9/site-packages/transformers/configuration_utils.py&quot;, line 427, in get_config_dict
    config_dict = cls._dict_from_json_file(resolved_config_file)
  File &quot;~/.local/lib/python3.9/site-packages/transformers/configuration_utils.py&quot;, line 510, in _dict_from_json_file
    text = reader.read()
  File &quot;/usr/lib/python3.9/codecs.py&quot;, line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

Any ideas what's causing this &amp; how to solve it?

It seems that @npit is right. The output of the convert_electra_original_tf_checkpoint_to_pytorch.py does not contain the configuration that I gave (hparams.json), therefore I created an ElectraConfig object -- with the same parameters -- and provided it to the from_pretrained function. That solved the issue.
