|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- lambada |
|
|
language: |
|
|
- en |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- text-generation-inference |
|
|
- causal-lm |
|
|
- int8 |
|
|
- PyTorch |
|
|
- PostTrainingStatic |
|
|
- Intel® Neural Compressor |
|
|
- neural-compressor |
|
|
--- |
|
|
# INT8 GPT-J 6B |
|
|
|
|
|
## Model Description |
|
|
GPT-J 6B is a transformer model trained using Ben Wang's [Mesh Transformer JAX](https://github.com/kingoflolz/mesh-transformer-jax/). "GPT-J" refers to the class of model, while "6B" represents the number of trainable parameters. |
|
|
|
|
|
This int8 PyTorch model is generated by [intel-extension-for-transformers](https://github.com/intel/intel-extension-for-transformers). |
|
|
| Package | Version | |
|
|
|----------------------|------------| |
|
|
| intel-extension-for-transformers| a4aba8ddb07c9b744b6ac106502ec059e0c47960 | |
|
|
| neural-compressor | 2.4.1 | |
|
|
| torch | 2.1.0+cpu | |
|
|
| intel-extension-for-pytorch | 2.1.0 | |
|
|
| transformers | 4.32.0 | |
|
|
|
|
|
## Evaluation results |
|
|
Evaluating the accuracy of the optimized model of gpt-j-6b using the lambada_openai dataset in lm_eval. |
|
|
|
|
|
| Dtype | Dataset | Precision | |
|
|
|------ |--------|--------| |
|
|
| FP32 |Lambada_openai | 0.6831 | |
|
|
| INT8 |Lambada_openai | 0.6835 | |
