---
tags:
- generated_from_trainer
model-index:
- name: gpt-4-est-base
  results: []
---

# gpt-4-est-base

A GPT model for Estonian, trained from scratch on 2.2 billion words (Estonian National Corpus + News Crawl + Common Crawl).

Model details:
- num. of layers: 12
- num. of heads: 12
- embedding size: 768
- context size: 1024
- total size: 118.68M params

Further details to be added soon.

### Framework versions

- Transformers 4.13.0.dev0
- Pytorch 1.10.0+cu102
- Datasets 1.15.1
- Tokenizers 0.10.3