pytorch zh transformers ZEN chinese license:apache-2.0
---
language: 
  - zh
license: apache-2.0

tags:
- ZEN
- chinese

inference: false

---
# Erlangshen-ZEN1-224M-Chinese, one model of [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM).

Erlangshen-ZEN1-224M-Chinese is an open-source Chinese pre-training model of the ZEN team on the [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM). IDEA-CCNL refers to the [source code of ZEN1.0](https://github.com/sinovation/ZEN) and the [paper of ZEN1.0](https://aclanthology.org/2020.findings-emnlp.425/), and provides the Chinese classification task and extraction task of ZEN1.0 effects and code samples. In the future, we will work with the ZEN team to explore the optimization direction of the pre-training model and continue to improve the effect of the pre-training model on classification and extraction tasks.

## Usage
There is no structure of ZEN1 in [Transformers](https://github.com/huggingface/transformers), you can run follow code to get structure of ZEN1 from [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM)

 ```shell
 git clone https://github.com/IDEA-CCNL/Fengshenbang-LM.git
 ```

## load model
```python

from fengshen.models.zen1.ngram_utils import ZenNgramDict
from fengshen.models.zen1.tokenization import BertTokenizer
from fengshen.models.zen1.modeling import ZenModel

pretrain_path = 'IDEA-CCNL/Erlangshen-ZEN1-224M-Chinese'

tokenizer = BertTokenizer.from_pretrained(pretrain_path)
model = ZenForSequenceClassification.from_pretrained(pretrain_path)
# model = ZenForTokenClassification.from_pretrained(pretrain_path)
ngram_dict = ZenNgramDict.from_pretrained(pretrain_path, tokenizer=tokenizer)

```

You can get classification and extraction examples below.

[classification example on fengshen]()


[extraction example on fengshen]()

## Evaluation

### Classification

|  model   | dataset  | Acc |
|  ----  | ----  | ---- |
| IDEA-CCNL/Erlangshen-ZEN1-224M-Chinese | Tnews | 56.82% |

### Extraction

|  model   | dataset  | F1 |
|  ----  | ----  | ---- |
| IDEA-CCNL/Erlangshen-ZEN1-224M-Chinese | OntoNote4.0 | 80.8% |


## Citation
If you find the resource is useful, please cite the following website in your paper.
```
@inproceedings{diao-etal-2020-zen,
    title = "ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations",
    author = "Diao, Shizhe and Bai, Jiaxin and Song, Yan and Zhang, Tong and Wang, Yonggang",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
    month = nov,
    year = "2020",
    address = "Online",
    pages = "4729--4740",
}
```