Training in progress, step 2500

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:171012e5b4def0bc073ed6ffc1b59ed030338c86f4a2bc801a88929f395604b6
+oid sha256:a7469f6ee953696180d23d25617c9fdd2e37f857039349f9db3edff5c808b3b5
 size 1262083569

wandb/run-20220202_140930-21azk117/files/output.log

@@ -2199,3 +2199,552 @@ Configuration saved in ./checkpoint-2000/config.json
 {'eval_loss': 0.47254303097724915, 'eval_wer': 0.40884505249761377, 'eval_runtime': 138.2935, 'eval_samples_per_second': 18.866, 'eval_steps_per_second': 0.268, 'epoch': 50.0}
 Model weights saved in ./checkpoint-2000/pytorch_model.bin
 Configuration saved in ./checkpoint-2000/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-1500] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 52%|██████████████████████████████████████████████████████████████████████████████████████▋                                                                              | 2100/4000 [5:56:32<5:50:22, 11.06s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 55%|██████████████████████████████████████████████████████████████████████████████████████████▋                                                                          | 2199/4000 [6:12:40<4:36:54,  9.23s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 57%|██████████████████████████████████████████████████████████████████████████████████████████████▊                                                                      | 2300/4000 [6:29:03<5:06:02, 10.80s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 60%|███████████████████████████████████████████████████████████████████████████████████████████████████                                                                  | 2400/4000 [6:45:05<3:23:23,  7.63s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 62%|███████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                             | 2500/4000 [7:01:26<4:34:00, 10.96s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 2609
+  Batch size = 72
+{'loss': 0.0731, 'learning_rate': 0.00038917712440717607, 'epoch': 62.5}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Configuration saved in ./checkpoint-2500/config.json
+{'eval_loss': 0.45677438378334045, 'eval_wer': 0.3824053452115813, 'eval_runtime': 136.9902, 'eval_samples_per_second': 19.045, 'eval_steps_per_second': 0.27, 'epoch': 62.5}
+Model weights saved in ./checkpoint-2500/pytorch_model.bin
+Configuration saved in ./checkpoint-2500/preprocessor_config.json

wandb/run-20220202_140930-21azk117/run-21azk117.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f7f400636d32d1d18a5befd3d42eef52cedfa83f9cdb0971a9331e2111cef2bb
-size 14549345
+oid sha256:8603ccd8e532bddc3921f83c49062eccf50e4e66433ed445af033912153dbe4b
+size 18190542