End of training

README.md

@@ -78,7 +78,7 @@ GPT2LMHeadModel(
 
 # Resource Usage
 
-- Max Train VRAM Use: 15.8655 GB
+- Max Train VRAM Use: 15.2844 GB
 - Available VRAM: 23.6429 GB
 - GPUs: 
   - 1x NVIDIA GeForce RTX 4090
@@ -115,7 +115,7 @@ GPT2LMHeadModel(
 <br/>
 
 # Train Dataset
-Trained on 521,346,966 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 521,334,462 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `990,000`
 - Subset: `20231101.en`
@@ -134,11 +134,7 @@ DistillationObjective(
         weight=0
     ),
     attn_loss_component=LossComponent(
-        weight=5,
-        loss_fn='raw_mse',
-        layer_mapper='layer-2',
-        norm='layernorm_teacher_only_affine',
-        projector='mlp_64_l3'
+        weight=0
     )
 )
 ```
@@ -165,14 +161,10 @@ The following hyperparameters were used during training:
         weight=0
     ),
     attn_loss_component=LossComponent(
-        weight=5,
-        loss_fn='raw_mse',
-        layer_mapper='layer-2',
-        norm='layernorm_teacher_only_affine',
-        projector='mlp_64_l3'
+        weight=0
     )
 )`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7efc6c1dab60>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7facb65ee170>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `distilbert/distilgpt2`
 - student_model_config: `None`
@@ -191,7 +183,7 @@ The following hyperparameters were used during training:
 - gradient_accumulation_steps: `1`
 - weight_decay: `0.0`
 - max_grad_norm: `1.0`
-- warmup_ratio: `0`
+- warmup_ratio: `0.0`
 - warmup_steps: `0`
 - gradient_checkpointing: `True`
 

logs/attn_weight=0/events.out.tfevents.1725646785.a7e428977e35

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:38cf51078356b96b7f32b5b97b17665b161655c3f6795053842fd60bd25928f9
-size 1910806
+oid sha256:87b77783df13afe858bfb41cf0e02bb58a15d941fc29fb263a464be88df196e1
+size 1970995

logs/attn_weight=0/events.out.tfevents.1725679647.a7e428977e35

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79190e6f08b94a33384ec604313e0d71556e27915728ed6df60d6072c9633990
+size 529

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:227716479b8200c41e9ec0b3caee412f709447e53ac129f1ff919dc0c1678c0d
+oid sha256:c50903859f256cc52993c2d269c80e12457fead0c3de901f87ec1ee2082ca68f
 size 163832792