Create README.md

README.md

@@ -0,0 +1,11 @@
+6.7m param GPT-J model.
+```
+n_positions - 128
+n_embd - 64
+n_layer - 4
+n_head - 8
+rotary_dim - 64
+tokenizer - gpt-j
+```
+
+Trained on 4,194,304 samples from the [c4](https://hf.co/datasets/c4) dataset, at a length of 128 tokens each, that comes out to 536,870,912 (0.53B) tokens seen during training. A batch size of 16 with 128 gradient accumulation steps was used, making the effective batch size 2048.