* GPT-J was trained on a TPU v3-256 for about five weeks
* OPT was trained on 992 NVIDIA 80GB A100 GPUs for about two months, and had to deal with several hardware failures over that time that required either manual or automatic restarts.
* Bloom was trained on the Jean Zay, which had 384 NVIDIA A100 80GB GPUs, and took about four months

The TPUs used by google are not for sale, and a single NVIDIA 80GB A100 GPU can cost 10,000$-20,000$, in addition to the extra components needed to wire everything together and cool everything. In order to train a model on the same time scale as those, if we assumed 300 A100 GPUs was enough, the cost for a facility to train them might be around 6 million$.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence